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EXECUTIVE  SUMMARY 


This  report  presents  the  results  of  a  study  of  various  instruction 
methodologies  currently  in  use  to  teach  people  how  to  access  and  search 
computer- stored  bibliographic-type  databases  in  an  online  mode.  The  study 
was  performed  to  provide  DTIC  management  with  information  about  practices 
and  experiences  for  DTIC  use  in  reaching  decisions  about  its  future  course 
of  action  with  respect  to  educating  clientele  in  DTIC’s  access  system  and 
online  databases. 

Discussions  with  experts  in  the  online  community,  including  members 
of  the  DTIC  staff,  led  to  the  selection  of  programs  of  the  following  organi¬ 
zations  for  examination: 

a.  Lockheed  Information  Retrieval  Service  (LIRS)  (2  programs) 

b.  University  of  Denver  Graduate  School  of  Librarianship  and 

Information  Management  (GSLIM) 

c.  National  Library  of  Medicine  (NLM) 

d.  University  of  Dayton  Research  Institute  (UDRI) 

e.  University  of  Pittsburgh  On-Line  Training  Center  (Pitt) 

f.  House  Information  Systems,  House  of  Representatives  (H.I.S.) 

g.  Congressional  Research  Service,  Library  of  Congress  (CRS) 
as  well  as  DTIC  for  comparison  purposes. 

A  set  of  criterion  variables  was  developed  by  which  the  instruction 
programs  were  analyzed  and  compared.  These  variables  address  four  major  area 

1.  Purposive  documentation 

goals,  objectives,  criteria 

2.  Education  program 

components,  duration(s),  availability,  location(s), 

participant  s 

3.  Resources 

human,  equipment,  facilities,  instruction  materials 

4.  Evaluation/Feedback  Mechanisms 


instruments,  tests,  follow-ups,  user  groups 


Most  programs  are  found  to  be  weak  in  purposive  documentation  that 
limits  the  bases  on  which  their  developers  can  evaluate  their  accomplishments. 
Five  of  the  programs  rely  on  lectures  as  the  principal  mode  of  knowledge 
transfer.  Pitt  and  H. I.S.  programs  integrate  lecturing  with  hands-on  practice. 
LIRS's  ONTAP  and  CRS  programs  provide  online  self-instruction.  Four  of  the 
programs  have  a  multi-course  structure,  offering  one  or  more  advanced  courses 
beyond  an  introductory  one.  No  course  other  than  the  academic  one  (GSLIM) 
is  longer  than  one  week  in  duration.  Only  LIRS's  ONTAP  program  is  substantially 
location  independent;  all  others  require  students  to  travel  to  either  a  single 
location  or  one  of  several  locations.  Although  six  of  the  courses  have  no 
course  fees,  the  high  indirect  costs  of  travel  probably  prevent  some  individuals 
needing  instruction  from  obtaining  it. 

Approximately  1  to  1^  months  appears  to  be  needed  by  an  instructor  to 
plan  a  program  and  prepare  printed  instruction  materials  for  each  content 
focus,  i.e.,  a  command  language  or  a  database.  Preparation  time  for  two  or 
more  foci  appears  to  be  cumulat:  .'e,  for  example,  2-4  months  for  a  command 
language  and  a  database.  Complex  databases  require  more  time,  for  example, 

3  months  for  a  UDRI  manual.  Time  needed  to  revise  a  program  and  its  instruc¬ 
tion  materials  can  vary  from  3  days  to  1  month  dependent  on  the  extent  of  the 
revision.  By  comparison,  instructors  can  spend  10  to  100  hours  to  prepare 
one  hour  of  computer-assisted  instruction. 

No  "best"  student/instructor  ratio  for  online  instruction  is  yet  known. 
Ratios  of  the  programs  studied  vary  from  3/1  to  20/1,  but  they  are  based  on 
the  number  of  terminals  available  for  hands-on  practice.  Only  the  Fitt, 

ONTAP,  and  CRS  programs  provide  each  student  with  a  terminal;  in  most  other 
programs,  two  people  share  a  terminal.  Irrespective  ot  the  instruction  metho¬ 
dology  employed,  all  programs  produce  printed  materials  for  students.  These 
vary  from  very  brief  guides  to  a  database  and  retrieval  options  (H.I.S.)  to 
replicas  of  entire  lecture  content  (NLM). 


Six  of  the  programs  distribute  questionnaires  to  students  at  the  end 
of  a  program  to  obtain  a  course  evaluation.  No  program  has  a  longer  term 
evaluation  mechanism,  for  example,  a  follow-up  six  months  after  the  course. 

The  customer  service  desks  maintained  by  HRS,  DTIC,  NLM,  and  H.I.S.  provide 
feedback  to  instructors,  but  only  for  NLM  is  this  direct  since  NLM  instruction 
staff  also  perform  service  desk  duties.  The  user  group  meetings  of  DTIC  and 
NLM  and  the  update  sessions  of  LIRS  are  other  opportunities  for  feedback. 

No  one  methodology  can  be  judged  better  than  the  others  since  they 
differ  in  purpose,  command  language,  databases,  and  target  audiences.  Printed 
manuals  or  wookbooks  appropriate  to  client  needs  are  seen  as  essential  for  at 
least  post-course  on-the-job  reference.  The  lecture  mode  of  knowledge  transfer 
is  by  no  means  obsolete,  but  it  is  recommended  that  other  media  be  incorporated 
with  it  when  they  can  more  effectively  convey  information.  For  example, 
slide/tape  and  videotape  modules  exemplify  the  "one  picture  is  worth  a  thousand 
words"  adage.  These  visual/sound  media  could  be  components  of  lectures  and/or 
offered  for  self-paced  instruction.  Hands-on  practice  is  a  sine  qua  non  for 
these  instruction  programs.  Practice  totaling  4  to  5  hours  per  student  is 
suggested  for  programs  having  the  objective  of  providing  the  student  with 
proficiency  to  do  online  searches  on-the-job  for  themselves  or  end-users. 

GRS's  computer-based  instruction  (CBI)  program  demonstrates  that  the 
CBI  methodology  can  be  used  to  convey  introductory-level  knowledge  of  a 
command  language  and  databases.  CBI  is  not  yet  hospitable  to  teaching  advanced 
search  complexities.  It  is  also  expensive.  The  LIRS  ONTAP  approach,  that 
uses  computers  in  a  more  limited  way,  is  worthy  of  further  study.  It  is  self- 
instructional  and,  with  proper  printed  materials,  it  might  be  able  to  effectively 
replace  up  to  85%  of  present  instruction  programs. 
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INTRODUCTION 

The  study  that  is  the  subject  of  this  report  was  commissioned  by  the 
Defense  Technical  Information  Center  (DTIC)  because  of  the  confluence  of  at 
least  three  factors: 

(1)  a  managerial  interest  in  periodically  reviewing  DTIC's  instruc¬ 
tion  program  for  its  online  clientele  for  adequacy  and 
appropriateness, 

(2)  growth  of  the  pre-October  1980  demand  for  instruction  that  was 
beginning  to  overload  the  resources  DTIC  had  allocated  to  the 
instruction  activity, 

(3)  commencement  of  dialup  online  access  to  DTIC's  databases  in 
October  1980  that  was  expected  to  introduce  greater  demands 
for  instruction  from  a  more  heterogeneous  clientele. 

DTIC  personnel  responsible  for  customer  service  are  members  of  the  by 

now  sizeable  "online  community."  This  community  consists  of 

—  "database  producers"  or  "suppliers",  organizations  that  produce 

databases  in  machine  ( computer)-readable  form, 

—  "database  vendors"  or  "search  services,"  organizations  that 

purchase  or  otherwise  acquire  the  machine- readable  databases, 
store  the  databases  in  a  computer  system  that  they  maintain, 
and  make  the  databases  available  to  user  organizations, 

—  "user  organizations,"  organizations  that  gain  access  to  the 

vendor-stored  databases  through  purchase  or  some  other 
arrangement  and  have  a  client  or  customer  relationship 
with  the  vendor, 

--  "communications  utilities,"  organizations  that  maintain  networks 
that  link  user  organizations'  terminals  to  vendors'  computers, 

—  organizations  of  many  types,  including  higher  education  institu¬ 

tions,  consultant  groups,  individual  consultants,  and  library 
consortia,  that  offer  education  programs  to  enable  the  users 
to  effectively  use  the  computer-stored  databases, 

--  publishers  and  authors  who  perform  a  documentation  and  communi¬ 
cation  service  for  the  community. 


A  considerable  amount  of  functional  overlap  may  exist  within  a  particular 


organization.  For  example,  database  producers  are  increasingly  undertaking 
the  task  of  providing  users  with  one  or  more  education/training  courses. 

DT1C  is  an  illustration  of  a  database  producer  that  is  also  a  vendor.  All 
vendors  provide,  directly  or  through  a  contractor,  some  level  of  education  to 
actual  and  prospective  users. 

The  various  education  programs  currently  being  offered  reflect  the 
offeror's  judgment  about  the  needs  of  their  user  groups  and  their  education 
responsibility  to  these  groups.  Sometimes,  perhaps  frequently  in  the  case  of 
database  producers,  the  education  program  represents  a  compromise  between  no 
program  and  a  greater  level  of  effort  that  the  database  producer  cannot  afford. 

The  use  of  computer- stored  databases  is  becoming  increasingly  complex 
for  users  primarily  because  of  the  increasing  numbers  of  databases  and  vendor 
systems  that  are  becoming  available.  Most  databases  that  had  been  produced 
in  printed  form  are  now  also  in  machine-readable  form  and  stored  in  one  or 
more  public  access  systems.  To  be  sure,  some  of  the  databases  (e.g. ,  those 
of  the  Chemical  Abstracts  Service  and  the  BioSciences  Information  Service) 
are  structurally  complex  and  subject  knowledge  facilitates  use  in  the  printed 
or  online  mode.  Aside  from  inherent  file  content  complexity,  the  growing 
need  of  users  to  make  selection  determinations  from  among  available  databases 
imposes  a  new  level  of  difficulty.  Vendors  are  perceiving  the  need  to  provide 
instruction  about  databases  applicable  to  particular  subject  areas.  At  least 
some  database  producers  appear  to  be  reassessing  their  educational  level  of 
effort  because,  if  users  lack  sufficient  knowledge  about  a  database,  they 
may  ignore  it  in  preference  to  one  or  two  others  they  are  familiar  with  despite 
their  unsuitability  to  the  information  need.  Since  payments  for  use  (i.e., 
fees,  royalties)  are  becoming  a  significant  source  of  income  for  database 
producers,  some  are  beginning  to  make  greater  investments  in  education  programs. 

Most  database  producers  and  vendors  have  chosen  to  develop  an  in-housc 
course  preparation  and  teaching  capability.  A  few  producers  arc  obtaining 
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pare  or  all  of  their  education  programs  through  contracts.  Staff  ceilings 
and  Office  of  Management  and  Budget  Circular  A-76  Revised  (March  29,  1979) 
are  turning  many  federal  agencies  including  DTIC  to  investigate  the  private 
sector  option  for  the  performance  of  particular  activities. 

This  study  has  examined  eight  instruction  programs  in  addition  to 
DTIC* s  to  give  DTIC  management  information  to  use  in  reaching  a  decision  on 
its  future  course  of  action  with  respect  to  educating  clients  in  the  utiliza 
tion  of  DTIC's  online  system  and  databases. 


STUDY  OBJECTIVES 


I _ L 

An  interagency  agreement  was  established  between  DTIC  and  the  Of  lice 
of  Personnel  Management,  Workforce  Effectiveness  and  Development  Group,  for 
performance  of  the  instruction  methodologies  study  by  an  OPM  specialist. 

The  author  is  a  member  of  the  online  community  who  has  participated  in  develop¬ 
ing  and  teaching  a  quarter- length  course  in  an  accredited  library  school. 

The  agreed  study  objectives  are: 

a.  To  identify,  analyze,  and  evaluate  training  programs  that 
utilize  different  modes  of  instruction  to  determine  the  mode 
or  combination  of  modes  most  productive  and  cost  effective 
for  DTIC*s  future  training  programs  for  users  of  the  Defense 
On-Line  System  (DROLS). 

b.  To  review  implications  of  the  instruction  methodologies  for 
DTIC  consideration  in  formulating  and  implementing  future 
education  programs. 

DTIC  identified  particular  types  of  education  methodologies  and  train¬ 
ing  programs  that  were  of  interest.  These  were: 

--  computer-assisted  instruction 

—  programmed  texts 

—  contractor-supplied  training 

--  audiovisual  modules 

--  self-teaching  modules 

--  lecture  augmented  by  printed  materials 

--  beginner/advanced/ ref resher  modules 

Within  fund  and  investigative  time  constraints,  it  was  agreed  that  up 
to  8  programs  that  exemplified  the  foregoing  methodologies  or  combinations 
of  them  would  be  reviewed,  compared  to  the  extent  possible,  and  assessed  tor 
applicability  to  DROLS. 
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STUDY  METHODOLOGY 

The  initial  phase  of  the  study  focused  on  selection  of  the  programs  to 
be  reviewed.  Preliminary  inquiries  were  made  by  telephone  about  programs 
known  to  DTIC  personnel  and  the  investigator.  These  inquiries  soughL  informa¬ 
tion  about  the  methodologies  employed.  In  several  instances  it  was  found 
that  a  reputation  for  use  of  a  methodology  other  than  the  lecture  mode  was 
in  error.  Telephone  inquiries  were  also  made  to  several  major  library  consortia; 
library  schools,  and  consultants  to  "brainstorm"  for  candidate  program  selec¬ 
tion.  Where  uncertainty  was  expressed  about  the  nature  of  a  program,  follow-up 
telephone  calls  were  made. 

It  became  apparent  that  the  lecture  mode  of  teaching  the  use  of  online 
databases  is  sufficiently  pervasive  that  a  modification  of  the  study  plan  had 
to  be  introduced.  The  initial  expectation  was  that  many  methodologies  could 
be  found  in  a  relatively  "pure"  form,  that  is,  used  by  themselves  not  in 
combination  with  others.  It  was  decided  to  accept  programs  that  used  either 
one  methodology  or  one  predominantly  or  a  variety  of  methodologies  such  that 
the  methodological  novelty  resided  in  the  mix  being  used.  Although  the 
lecture  mode  is  pervasive,  it  was  instructive  to  find  that  lecture  alone  is 
being  phased  out  as  an  instructional  methodology. 

Accordingly,  the  following  programs  were  chosen  for  review: 

-  for  computer-based  instruction  (CBI,  frequently  used  interchange¬ 
ably  with  "computer-assisted  instruction"  or  CAI) 

Congressional  Research  Service  (CRS)*s  SCORPIO  program 

-  for  an  integration  of  several  media  and  for  a  contractor- 
supplied  program 

University  of  Dayton  Research  Institute  (UDRI)'s  FTD 

program 

-  for  a  self-instruction  program 

Lockheed  Information  Retrieval  Service  (LIRS)fs  ONTAP 


program 


-  9  - 


-  for  Learning  predominantly  through  a  hands-on  practice  program 

House  Information  Systems  (H.I.S.)'s  multi-database  program 

-  for  a  50/50  lecture/hands-on  program 

University  of  Pittsburgh  On-Line  Training  Center  (Pitt)'s 
program 

-for  a  reasonably  typical  lecture-based  academic  program 

University  of  Denver  Graduate  School  of  Librarianship 
and  Information  Management  (GSLIM)'s  course 

-  for  a  mostly  lecture-based  training  program 

LIRS's  seminar  programs 

-  for  a  federal  agency  analog  to  DTIC'  s  DRQLS  program 

National  Library  of  Medicine  (NLM)'s  MEDLINE  program 

Additionally,  like  DTIG,  CRS,  H.I.S.,  and  NLM  are  database  producers. 
Like  DTIC,  CRS,  LIRS,  H.I.S,  and  NLM  have  a  vendor  or  vendor-type  relationship 
with  user  organizations.  Other  methodological  characteristics  of  these  programs 
are  discussed  in  the  next  section  headed  Study  Findings. 

Contact  persons  for  the  programs  were  identified  and  telephoned.  All 
individuals  expressed  a  willingness  to  be  interviewed  and  to  share  their 
education  program  experiences  with  the  investigator. 

In  all  instances,  information  about  the  programs  was  obtained  by  inter¬ 
view.  Interviews  were  conducted  at  the  education  sites  for  all  organizations 
except  LIRS.  The  investigator  had  attended  two  of  LIRS's  recent  programs. 

For  LIRS,  the  contact  person  was  interviewed  by  telephone.  For  the  interviews, 
an  interview  schedule  was  developed,  field  tested,  and  modified.  The  final 
schedule  is  shown  in  Appendix  A.  In  most  instances,  the  interview  schedule 
was  sent  to  the  contact  person  before  the  interview  was  held.  In  all  instances, 
contact  persons  were  called  on  one  or  more  occasions  after  the  interview  for 
additional  or  verifying  information. 

To  comparatively  analyze  the  information  collected,  a  set  of  program 
characteristics  or  features  was  developed.  These  appear  in  the  left  column 
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of  Table  I  (page  68  ).  From  an  evaluation  methodology  viewpoint,  these 
characteristics  can  be  used  as  criterion  variables.^"  They  are  employed  in 
this  study  as  the  basis  for  a  comparative  analysis  and  assessment  of  education 
programs  and  methodologies.  It  should  be  noted  that  assessments  do  not  have 
to  be  value  loaded  (i.e.,  good,  bad).  Rather,  assessments  can  identify 
presence  or  absence,  or  degree  of  presence,  without  a  value  judgment.  Alter¬ 
natively,  the  characteristics  can  be  value  loaded.  For  example,  an  evaluator 
may  predetermine  that  a  student/instructor  ratio  in  the  range  of  3  to  5  is 
excellent,  6  to  9  is  very  good,  10  to  15  is  acceptable,  etc.  When  ratio  data 
are  collected  for  particular  programs,  they  can  be  rated  in  terms  of  the 
pre-established  scales. 

A  departure  from  the  original  study  methodology  was  introduced  about 
a  month  after  the  study  commenced.  It  had  been  desired  to  obtain  assessments 
of  the  effectiveness  of  education  programs  from  student  participants.  It 
began  to  become  evident  that  systematic  access  to  student  recipients  of  various 
programs  was  not  readily  attainable.  Consequently,  a  substitute  approach  to 
student  information  was  attempted  with  a  questionnaire.  The  3-page  question¬ 
naire  is  presented  in  Appendix  A  in  photoreduced  form.  The  questionnaire 
was  promoted  and  distributed  by  the  investigator  at  two  sessions  of  the 
Annual  Meeting  of  the  American  Society  for  Information  Sciences  in  October 
1980  and  at  a  meeting  of  the  On-Line  Users  Group  in  Washington,  D.C.  It  was 
also  disseminated  by  the  University  of  Dayton  and  the  University  of  Pittsburgh 
contact  persons  in  their  respective  locations.  The  total  return  was  16  ques¬ 
tionnaires  of  about  120  distributed,  a  disappointing  13%. 

Information  from  the  interviews  and  the  questionnaires  was  aggregated 
and  analyzed.  During  the  interviews,  contact  persons  were  asked  for  copies 
of  their  training  materials.  Not  all  training  materials  were  obtained  (e.g. , 
Chemical  Abstracts'  ONTAP  materials).  Available  materials  were  examined  as 
part  of  the  analysis  to  produce  this  report. 
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STUDY  FINDINGS 

This  section  is  subdivided  into  two  parts.  The  first  part  presents 
brief  descriptions  of  the  instruction  programs  selected  for  examination.  The 
second  part  contains  an  assessment  of  the  programs  oriented  to  assist  a  manager 
in  reaching  a  decision  about  appropriate  methodologies  for  his/her  organiza¬ 
tion's  clientele  and  databases. 


FINDINGS,  PART  I  -  INSTRUCTION  PROGRAM  DESCRIPTIONS 


Information  for  each  of  nine  programs  is  presented  in  the  following 

format : 

Type  of  Instruction 

Principal  mode  of  delivery 
Auxiliary  modes  of  delivery 
Organization  providing  instruction 
Name  and  address 
Contact  person(s) 

Telephone 

Type  of  organization  (i.e.,  public,  private,  vendor, 
database  producer,  etc.) 

Recipients  of  instruction 

Class  size(s) 

Location( s)  of  instruction 

Frequency  of  offering/availability 

Brief  description  of  nature  of  instruction 

Improvements  desired/planned 

References 

Additionally,  for  several  programs,  a  section  entitled  Background  has 
been  inserted  immediately  before  Brief  description  of  nature  of  instruction. 
For  these  programs,  it  was  believed  important  to  understanding  to  include 
either  the  historic  development  of  the  program  or  information  about  the 
databases  taught  in  the  program. 


For  the  last  data  entry.  References,  citations  are  given  for  reports 
and/or  published  papers  given  to  the  investigator  by  the  contact  person.  No 
literature  search  about  the  programs  was  conducted. 

The  program  descriptions  are  presented  in  this  section  by  a  Type  of 
Instruction  arrangement.  The  order  is  from  the  more  traditional  instruction 
methodologies  to  the  newer  methodologies.  By  organization,  the  pages  that 
follow  contain  descriptions  for  the  programs  of: 

Lockheed  Information  Retrieval  Service  (LIRS) 

University  of  Denver  Graduate  School  of  Librarianship 
and  Information  Management  (GSLIM) 

Defense  Technical  Information  Center  (DTIC) 

University  of  Dayton  Research  Institute  (UDRI) 

University  of  Pittsburgh  On-Line  Training  Center  (Pitt) 

House  Information  Systems  (H»I.S.) 

LIRS  ONTAP 

Congressional  Research  Service  (CRS) 

The  current  program  of  DTIC  is  included  to  provide  a  baseline  for  DTIC 


management 


Type  of  Instruction 

Principal  mode  of  delivery:  Lecture 
Auxiliary  modes  of  delivery: 

For  introductory  course:  Printed  workbook, 

Online  demonstrations, 

Online  practice 

For  subject  seminars  (advanced  courses)  and  refresher: 

Printed  workbook 

For  updates:  Sample  searches  handout 
Online  demonstration 

Organization  Providing  Instruction 

Lockheed  Information  Retrieval  Service  (LIRS) 

3460  Hillview  Avenue 
Palo  Alto,  CA.  94304 

Instructors 

Sophie  K.  Hudnut,  DIALOG  Training  Coordinator 
Staff  of  instructors,  number  not  disclosed 

(415)  858-3810;  (800)  227-1927  (except  CA);  (800)  982-5838  (in  CA) 

Type  of  Organization 

commercial  online  services  vendor 
also  contractor  supplying  instruction 

Recipients  of  Instruction 

primarily  DIALOG  users  and  prospective  users;  no  restrictions 
Class  size(s) 

Introductory  course:  up  to  12 
Subject  seminars:  up  to  25 
Refresher:  up  to  25 
Updates:  up  to  50 

Location(s)  of  Instruction 

Introductory  course:  field  offices,  major  cities  in  U.S. 

Refresher,  subject  seminars:  same  as  above 

Updates:  frequently  in  a  city  back-to-back  with  a  major  professional 
society  meeting 

Also  customer  locations,  see  Brief  Description  below 
Duration(s)  of  Instruction 

Introductory  course:  H  days  Advanced  chemical:  1  day 

Subject  seminars,  refresher:  \  day  Updates:  2  days 


Introductory  course:  almost  weekly  at  Palo  Alto,  Chicago,  New  York,  and 

Washington,  D.C.  field  offices 
several  times  per  year  in  each  of  most  major  U.S.  cities 

Subject  seminars,  refresher:  8  per  quarter  nationwide,  thus  each  offered 

about  3  times  per  year  somewhere  in  the  U.S, 

Advanced  chemical: 

Updates:  2  to  3  annually 

Brief  Description  of  Nature  of  Instruction 

This  instruction  program  focuses  solely  on  one  commercial  vendor's 
access  system,  Lockheed's  DIALOG,  and  on  the  databases  available  through  the 
system.  If  it  can  be  said  that  there  is  an  "official"  instruction  program  for 
DIALOG,  this  is  it  since  the  program  is  produced  and  provided  by  DIALOG  staff. 

LIRS's  position  on  instruction  has  changed  since  DIALOG  first  went 
public.  It  was  initially  thought  that  education  programs  would  be  developed 
by  educational  institutions  and  database  producers.  The  very  slow  entry  of 
these  organizations  into  the  education/training  field,  customer  need  for 
training,  and  LIRS  receptivity  to  providing  assistance  gradually  brought  LIRS 
to  its  present  position  of  taking  responsibility  for  training  in  the  use  of 
its  system  and  for  providing  at  least  an  introductory  understanding  of  its 
databases. 

Five  categories  of  instruction  are  provided: 

(1)  an  introductory  course  called  "System  Seminar" 

(2)  a  refresher  course  called  "System  Refresher" 

(3)  advanced  courses  called  "Subject  Seminars" 

(4)  one  more  intensive  advanced  course  called  "Advanced  Chemistry" 

(5)  updates  that  LIRS  doesn't  classify  as  courses  but  are  tutorial 

The  System  Seminar  gives  reasonably  well  balanced  instruction  in 
(a)  the  access  language  (DIALOG),  (b)  search  strategy  and  tactics,  (c)  data¬ 
base  structure,  and  (d)  integration  of  these  elements.  Illustrative  uses  of 
the  access  language  and  illustrative  search  problems  demonstrate  the  content 
of  a  variety  of  databases  and  retrieval  possibilities  from  them.  Considerable 
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emphasis  is  given  to  strengths  and  pitfalls  of  various  search  strategy 
specifications.  In  the  workbook  that  students  keep,  various  aspects  of 
search  outcomes  are  highlighted  and  annotated. 

The  amount  of  hands-on  practice  that  students  receive  in  the  introduc¬ 
tory  course  varies  with  the  availability  of  terminals  at  the  course  location. 
About  3  to  4  hours  of  online  time  are  allocated  per  course.  When  it  is  given 
in  Palo  Alto  with  6  terminals  available,  each  student  may  receive  up  to  1  hour 
of  practice  (individuals  working  at  the  different  terminals  simultaneously). 
When  a  single  terminal  is  available,  each  student  may  receive  only  15  minutes 
of  practice;  some  may  get  less.  When  the  course  is  given  at  a  customer's 
location  (see  below),  the  amount  of  practice  is  determined  by  the  amount  of 
time  the  customer  pays  for. 

LIRS  currently  offers  14  courses  beyond  the  introductory: 

-  Subject  Seminars  in  Biosciences,  Business,  Chemistry,  Excerpta 

Medica,  Government  Documents,  Humanities,  Legal  Information, 
Nonbibliographic  Databases,  Patents,  Science/Technology,  and 
Social  Sciences 

-  a  Seminar  in  Search  Strategy 

-  a  Refresher  Seminar 

-  an  Advanced  Chemistry  Seminar. 

An  effort  is  made  to  present  each  of  the  Subject  Seminars  and  the  Refresher 
in  each  part  of  the  U.S.  (to  minimize  travel  expenses  and  inconvenience)  once 
annually,  although  the  decision  for  a  particular  seminar  in  a  particular 
location  is  undoubtedly  also  determined  by  expressed  customer  need.  The 
Seminars  have  neither  online  demonstrations  nor  online  practice  on  the  assump¬ 
tion  that  students  have  this  experience  and  primarily  need  a  better  knowledge 
of  use  of  various  files  in  a  subject  field.  Each  of  the  workbooks  prepared 
for  the  seminars  presents  a  discussion  of  advanced  features  of  DIALOG  and 
annotated  illustrations  of  subject  searches. 
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LIRS  will  additionally  provide  any  of  the  training  courses  at  a 
customer's  location  for  a  fee  of  $300  for  a  half  day,  $375  for  2  half-days, 

$500  for  1^  days,  and  $575  for  2  days.  LIRS  furnishes  one  instructor  per 
course  regardless  of  the  location.  Each  instructor  is  a  specialist  in  one 
or  more  subject  areas.  Lately,  two  or  more  courses,  the  Introductory  course 
and  a  Subject  Seminar,  are  being  offered  back-to-back  at  a  location  to  reduce 
expenses.  A  salutary  aspect  of  this  practice  is  that  a  Subject  Seminar  can 
be  given  with  fewer  than  10  students  since  the  major  expense  has  been  covered 
by  the  introductory  course. 

The  updates  are  viewed  by  Lockheed  as  opportunities  for  experienced 
searchers  to  discuss  problems  and  strategies  among  themselves  and  with  LIRS 
personnel  and  representatives  of  database  producers.  For  example,  the  Fall 
1980  Update  held  in  San  Francisco  before  the  Online  ’80  meeting  featured  7 
subject  sessions,  each  focused  around  5  search  topics.  Arrangements  were  made 
for  relevant  database  producers  to  prepare  and  discuss  the  strategies  they 
formulated  for  the  topics.  The  Update  also  included  sessions  about  new  DIALOG 
databases,  the  DIALORDER  procedure,  and  LIRS's  private  files  service;  a  "What's 
New  with  DIAI/3G?"  summary  by  the  DIALOG  director;  and  a  database  exhibit  at 
which  producer  brochures  were  displayed. 

Improvements  Desired/Planned 

The  Training  Coordinator  views  education  as  an  evolving  process  and 
states  that  training  materials  are  under  continuous  review  for  revision  as 
needed.  In  September  1980,  LIRS  announced  a  major  revision  in  its  database 
documentation  ("Guide  to  DIALOG  -  Databases").  LIRS  is  reissuing  this  publi¬ 
cation  as  separate  chapters  and  it  is  eliminating  preliminary  versions  of 
chapters  that  were  heretofore  distributed  as  separates  until  a  sufficient 
number  of  chapter  texts  could  be  assembled  for  a  looseleaf  volume  of  the 


"Guide." 


A 
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In  1980,  several  new  Subject  Seminars  were  introduced  and  the  pattern 
is  likely  to  continue  in  1981.  LIRS  courses  may  also  be  offered  in  more 
locations  in  the  U.S.  An  item  in  LIRS*s  November  1980  newsletter,  Chronolog, 
encouraged  DIALOG  users  to  become  host  institutions  for  DIALOG  courses.  An 
inducement  is  two  free  registrations  for  each  seminar  hosted  or  an  equivalent 
amount  of  connect-time  credit.  The  items  specified  the  need  for  three  tele¬ 
phone  lines  (for  3  terminals)  for  the  Introductory  course. 


References 

"DIALOG  System  Seminar" 

"Chemical  Information  Seminar",  "Business  Seminar,"  etc.  for  each  subject 
seminar 

"Guide  to  DIALOG  Searching" 

"Guide  to  DIALOG  -  Databases"  -  looseleaf  set  of  chapters,  each  averaging 

about  30  pages  and  providing  a  field-by-field  description  of  the  database 
as  stored  in  the  Lockheed  system  with  several  sample  searches 
"Chronolog"  monthly  newsletter 

Bluesheets,  two  page  summaries  of  the  contents  of  fields  of  each  database 
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Type  of  Instruction 

Principal  mode  of  delivery:  Lecture 
Auxiliary  modes  of  delivery:  Printed  material, 

Online  demonstration. 

Hands-on  practice 

Organization  Providing  Instruction 
University  of  Denver  (GSLIH) 

Graduate  School  of  Librarianship  &  Information  Management 
Denver,  CO.  80208 

Instructors 

James  K.  Foyle,  Anindya  Bose  (Rowena  W.  Swanson  to  March  j.980) 

(303)  753-2557 

Type  of  Organization 

private,  non-profit,  academic  institution  (graduate  school) 

Recipients  of  Instruction 

graduate  students  in  library/inf ormation  science 

Class  Size 

20  to  40 

Location  of  Instruction 

classroom  and  terminals  laboratory 

Duration  of  Instruction 

9  to  11  weeks  (4  hrs/wk  lecture,  2-5  hrs/wk  lab) 

Frequency  of  Offering/Availability 
quarterly 

Brief  Description  of  Nature  of  Instruction 

This  description  pertains  to  instruction  that  was  provided  in  1979 
and  early  1980.  Except  for  the  amount  of  hands-on  practice  time  allocated 
to  students,  the  methodology  is  typical  of  academic  instruction  programs  of 
the  time.  It  is  believed  that  university  programs  are  increasingly  providing 
the  level  of  hands-on  practice  available  at  GSLIM  and  perhaps  more. 

The  program  presented  instruction  in  two  systems,  Lockheed's  DIALOG 
and  SDC's  ORBIT.  In  both  instances,  all  features  of  the  command  languages 


were  taught.  Contents  of  databases  were  not  taught  on  the  assumption  that 
students  obtained  information  about  some  of  them  in  earlier  courses.  Data¬ 
base  design  and  loading  into  the  respective  computer  systems  were  explained 
in  relation  to  their  effect  on  search  strategy  development  and  retrieval. 

Most  lecture  periods  included  extensive  online  demonstrations  of  features 
selected  by  both  the  instructor  and  students.  Within  limits,  demonstrations 
were  performed  by  students  as  well  as  the  instructor.  Printed  manuals  distri¬ 
buted  to  students  contained  illustrations  of  online  sessions  that  were  used 
when  computer  systems  went  down. 

Student  performance  was  monitored  by  written  tests,  one  for  each 
system,  and  by  search  results  obtained  for  3  projects.  Two  projects  involved 
2  or  3  person  teams  who  selected  their  search  questions  from  a  collection  of 
real  inquiries  obtained  from  local  area  libraries.  The  third  project  was  an 
individual  one  chosen  by  the  student  and  approved  by  the  instructor.  Students 
selected  2  or  3  of  the  databases  most  appropriate  to  the  search  question  to 
search.  Neither  classroom  demonstrations  nor  student  practice  was  restricted 
to  one,  two,  or  a  few  files.  Students  discussed  their  search  results  and 
problems  encountered  in  the  projects  in  class  so  that  various  experiences 
could  be  shared.  A  total  of  three  30  to  40  minute  online  practice  periods 
were  allotted  to  each  student  for  project  work. 

The  laboratory  is  equipped  with  3  keyboard-CRT-printer  terminals  of 
different  manufacturers  for  breadth  of  experience.  It  also  contains  most  of 
the  manuals,  thesauri,  and  search  aids  that  have  been  published  for  online 
databases,  including  materials  for  systems  not  taught  during  the  course  (e.g. , 
MEDLARS).  Students  typically  would  spend  2  to  4  hours  of  their  own  time  weekly 
in  the  laboratory  working  with  the  materials  for  their  projects. 

Improvements  Desired/Planned 

Most  student  requests  were  for  additional  hands-on  practice. 

Faculty  desired  additional  terminals  (at  least  two)  of  different 
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manufactures  to  give  students  broad  experience  for  the  job  market.  Existing 
terminals  operated  at  300  baud.  At  least  one  1200-baud  terminal  was  wanted. 

To  provide  reasonably  thorough  comprehension  of  all  --  not  only  the 
basic  --  elements  of  the  DIALOG  and  ORBIT  command  languages,  no  reduction  in 
exposure  time  to  the  systems  was  deemed  possible.  Nevertheless,  faculty 
wanted  to  provide  learning  opportunities  for  the  BRS  system  and  the  New  York 
Times  Information  Bank;  MEDLINE  is  briefly  presented,  without  hands-on 
practice,  in  a  medical  bibliography  course. 

No  funds  were  available  for  additional  equipment  or  online  time.  Some 
thought  was  given  to  providing  instruction  on  other  systems  in  short-course 
formats  (1-2  weeks  of  intensive  instruction  and  practice  with  a  different 
pricing  structure).  Advanced  short-course  instruction  that  might  focus  on 
subject  areas  and  databases  for  them  was  also  being  considered  for  future 
development. 


i 
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Type  of  Instruction 

Principal  mode  of  delivery:  Lecture 
Auxiliary  modes  of  delivery:  Hands-on  practice, 

Printed  materials. 

Flip  charts 

Organization  Providing  Instruction 

Defense  Technical  Information  Center  (DT1C) 

Defense  Logistics  Agency 
Cameron  Station,  Bldg.  5 
Alexandria,  VA.  22314 

Instructor 

James  de  Persis 
(202)  274-7206 

Type  of  Organization 

public,  federal  agency,  database  producer 

Recipients  of  Instruction 

For  dedicated  system:  85%  federal  agencies  staff, 

15%  DOD  contractor  personnel 

For  dialup  system:  same  as  above,  70%,  contractor  personnel  expected, 

mix  of  clerical,  library,  sci/tech  staff  members 

Class  Size 

8  to  10 

(50  people  in  a  2-day  dialup  session  in  Los  Angeles  in  December  1980) 

Location! s)  of  Instruction 
Training  room  in  DTIC 

Training  facilities  provided  by  organizations  in  the  field 
Duration! s)  of  Instruction 

For  dedicated  system:  Introductory  course:  5  days  (majority  of  courses) 

8  days,  10  days  (rarely) 

Intro,  in  field:  5  days 
Intermediate  course:  3  or  5  days 

None  held  in  field  through  1980 
Advanced  course:  3  days 

None  held  ir  field  through  1980 

For  dialup  system:  Introductory  course:  2,  3,  or  5  days  planned 

(instruction  commenced  in  late  1980) 
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Frequency  of  Offering/Availability 

For  dedicated  system:  Introductory:  at  least  one  monthly  at  DTIC 

In  field  when  number  of  requests  warrants  expense 
Intermediate,  Advanced:  on  an  as  requested  basis, 
as  number  of  requests  reaches  6  to  10 
For  dialup  system:  Pattern  not  yet  established 

Background 

DTIC  produces  five  databases: 

a.  Technical  Reports  (TR)  file 

Contains  bibliographic  data;  abstract,  subject  terms,  and  copy  prices 
(DD  Form  1473  information)  for  scientific  and  technical  reports  pro¬ 
duced  by  DOD  agencies  and  their  contractors.  The  online  file  contains 
surrogate  information  for  documents  acquired  by  DTIC  during  the  past 
10  years.  (Information  for  documents  acquired  earlier  is  not  main¬ 
tained  online. ) 

b.  Current  Technical  Reports  (CF)  file 

Contains  DD1473  information  for  documents  to  be  announced  in  the  next 
issue  of  the  DTIC  publication.  Technical  Abstracts  Bulletin  (TAB). 

After  publication  of  the  TAB  issue,  the  data  are  transferred  to  the 
TR  file. 

c.  Work  Unit  Information  (WU)  file 

Contains  management  and  technical  status  data  (DD  Form  1498  information) 
for  activities  being  performed  in  DOD  Research  and  Development  pro¬ 
grams.  A  "work  unit"  is  the  smallest  segment  into  which  R&D  efforts 
are  divided;  for  example,  a  DOD  project  may  consist  of  a  set  of  tasks 
each  of  which  is  composed  of  several  work  units.  The  WU  file  is  a 
"full  text"  file  in  the  sense  that  it  contains  all  of  the  information 
reported  on  DD1498;  this  includes  agencies  and  individuals  doing  the 
work,  contract/grant  and  funding  data,  a  description  of  work  objectives 
and  content,  subject  terms,  identification  numbers  (program  element, 
etc.),  and  security  limitations. 

d.  Program  Planning  (PP)  file 

Contains  management,  fiscal  and  technical  status  data  (DD  Form  1634 
information)  for  DOD  R&D  programs  at  the  project  and  task  levels. 

An  R&D  "project"  consists  of  one  of  several  related  tasks  directed 
to  the  development  of  equipment,  systems,  components,  techniques,  or 
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to  the  exploration  ol  a  new  lield  of  knowledge.  The  PP  is  a  "lull 
text"  file  in  the  sense  that  it  contains  all  of  the  information 
reported  on  DD1634;  this  includes  agencies  and  individuals  engaged 
in  the  project,  fiscal  data  and  fund  resource  projections,  a  des¬ 
cription  of  work  objectives,  plans,  and  progress,  identification 
numbers,  and  security  information. 

e.  Independent  Research  (IRi  file 

Contains  planning  and  technical  information  for  non-DOD  sponsored 
projects  that  private  organizations  voluntarily  report  to  DOD  on 
DDC  Form  271.  The  projects  are  within  DOD  R&D  areas  of  interest. 

They  are  components  of  corporate  research  programs  directed  toward 
the  development  of  future  technologies,  systems,  and  hardware. 

The  data  are  proprietary  and  exempt  from  disclosure  under  the 
Freedom  of  Information  Act,  subsection  (b),  5  USC  552.  The  file 
is  accessible  only  to  DOD  and  other  federal  agencies  that  have 
been  certified  to  use  it.  Certification  is  further  limited  to 
need-to-know  fields  of  interest  and  appropriate  security  classifica¬ 
tions. 

Instruction  on  the  IR  file  is  only  included  in  the  10-day  Intro¬ 
ductory  course.  Participation  is  limited  to  individuals  and 
agencies  that  have  received  the  necessary  access  certifications. 

Brief  Description  of  Nature  of  Instruction 

Until  the  Fall  of  1980,  the  number  of  organizations  that  used  DTIC* s 
online  databases  was  limited  to  approximately  100  because  they  had  to  obtain 
direct  line  access  (dedicated  terminals).  Only  UNIVAC  100  or  200  terminals 
were  appropriate.  In  October  1980,  dialup  access  became  available  for  which 
there  are  no  terminal  restrictions.  This  expansion  of  databases  availability 
is  expected  to  significantly  increase  the  demand  for  all  levels  of  instruc¬ 
tion. 

For  direct  line  clients,  the  training  methodology  in  DTIC* s  Cameron 
Station  facility  consists  of  lecture  extensively  alternated  with  hands-on 
practice.  Three  keyboard-CRT-printer  terminals  are  in  the  training  room. 

The  instruction  pattern  is  to  subdivide  each  morning  and  afternoon  into  lecture 


-  24 

and  online  segments.  Both  lecture  and  student  preparation  for  practice 
require  frequent  reference  to  DTIC's  printed  search  aids.  The  instructor 
provides  sets  of  8  to  12  questions  that  are  to  be  searched  for  each  practice 
session.  Online  practice  is  performed  by  2  or  3  person  student  teams;  the 
individuals  take  turns  at  the  keyboard.  Students  use  the  approximately  20 
feet  of  chalkboard  in  the  room  to  record  results  of  some  of  the  searches  that 
are  then  discussed  at  the  beginning  of  the  next  lecture  period. 

The  5-day  Introductory  course  is  the  most  often  given.  It  covers  the 
command  language  and  the  TR  and  WU  databases.  The  8-day  course  adds  instruc¬ 
tion  for  the  PP  database.  The  10-day  course  includes  the  IR  database;  it  is 
infrequently  held  because  of  the  limited  clientele  for  this  file. 

The  Intermediate  and  Advanced  courses  have  only  been  presented  at 
DTIC's  facility.  The  Intermediate  course  has  been  held  for  students  of  the 
5-day  Introductory  course.  The  3-day  Intermediate  course  covers  a  review  of 
the  TR  and  WU  databases  and  training  in  the  PP  database.  The  5-day  Inter¬ 
mediate  course  includes  the  IR  database.  The  Intermediate  course  is  thus 
primarily  a  time  extension  of  the  Introductory  course. 

The  3-day  Advanced  course  is  held  mainly  to  provide  instruction  on 
changes  and  enhancements  to  the  system.  For  example,  the  Advanced  course 
most  recently  held  covered  text  searching  that  was  introduced  into  the  system 
in  1979.  Additionally,  the  course  is  a  forum  for  user  questions  and  problems. 

Only  5-day  Introductory  courses  have  been  held  for  direct-line  clients 
in  the  field.  The  coverage  is  essentially  the  same  as  for  the  course  at  DTIC 
except  that  only  one  terminal  is  typically  available  for  hands-on  practice 
rather  than  DTIC's  three. 

Instruction  for  dialup  clients  held  through  December  1980  introduced 
modifications.  The  duration  for  the  Introductory  course  was  shortened  to 
2  or  3  days  and  most  of  the  hands-on  practice  was  eliminated.  In  the  only 
experimental  3-day  course  held,  only  the  command  language  and  a  limited 
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explanation  oE  the  TR,  WU,  and  PP  databases  were  covered  with  no  hands-on 
practice.  At  a  2-day  dialup  course  in  Los  Angeles,  50  people  arrived  tor 
training;  only  2  terminals  were  available  and  no  screen  monitors  tor  audience 
viewing  of  demonstrations. 

All  students  are  given  certificates  regardless  of  whether  they  attended 
a  10-day  or  a  2-day  course.  Students  complete  a  DDC  Form  345,  a  course  evalu¬ 
ation  form,  that  is  anonymous.  About  50%  of  the  forms  are  estimated  to  contain 
constructive  information.  The  current  teaching  format  of  half  lecture,  half 
practice  each  morning  and  afternoon  emanates  from  student  requests.  The 
organization  of  the  October  1980  "Self-Training  Manual"  also  incorporates 
student  recommendations  that  instructional  text  be  arranged  by  database  rather 
than  by  system  commands.  Parenthetically,  this  text  is  designed  to  assist 
individuals  who  have  taken  DTIC's  courses  to  train  other  personnel  in  their 
organization  who  will  not  be  taking  DTIC's  courses. 

Improvements  Desired/Planned 

DTIC  is  planning  a  major  revision  in  its  training  programs.  It  commissioned 
a  study  of  trairing  methodologies  to  obtain  data  for  a  decision  on  its  future 
training  program  strategy. 

The  dialup  service  is  likely  to  appreciably  increase  client  demand  for 
training.  Since  the  DTIC  system  can  separate  access  to  the  classified  portions 
of  its  files  from  the  unclassified,  and  since  the  WU  and  PP  databases  are  not 
available  from  any  other  online  service,  popularity  of  these  files  can  be 
expected  to  grow.  (The  unclassified  portion  of  the  TR  file  is  contained  in 
the  database  of  the  National  Technical  Information  Service  that  is  publicly 
available.)  It  is  also  anticipated  that  the  dialup  clientele  will  differ  to 
some  extent  from  that  for  direct-line  service.  Dialup  users  will  probably 
consist  mainly  of  private- sector  organizations  with  a  greater  heterogeneity  of 
information  needs. 


DTIC  conducts  five  annual  user  group  meetings,  four  at  regional  loca- 
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tions  (e.g.,  California,  Midwest,  Florida,  Northeast)  and  one  at  DTIC  in  the 
Fall.  Attendees  at  the  October  1980  DTIC  meeting  strongly  recommended  that 
one  day  of  advanced  training  be  held  as  part  of  each  regional  meeting  program. 
Implementation  of  this  recommendation  is  planned.  Coverage  will  include  dis¬ 
cussions  of  user  experiences  and  problems  as  well  as  information  about  system 
changes. 

References 

"Self-Training  Module  for  Users  of  the  Defense  RDT&E  On-Line  System," 
September  1979. 

"Self-Training  Manual  for  Users  of  the  Defense  RDT&E  Diverse  Dial-Up  On-Line 
System,"  October  1980 

"Defense  RDT&E  On-Line  System,  Retrieval  Mini-Manual,"  December  1979. 
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Type  of  Instruction 

Principal  mode  of  delivery:  Lecture 

Auxiliary  modes  of  delivery:  Computer-based  instruction, 

Workbook, 

Hands-on  practice, 

Other  printed  materials 

Organization  Providing  Instruction 

National  Library  of  Medicine  (NLM) 

Bibliographic  Services  Division 
MEDLARS  Management  Section 
8600  Rockville  Pike 
Bethesda,  Maryland  20014 

Instructors 

Four  full-time  staff  (including  Patricia  E.  Healy  and  Sheila  Proudman 
who  were  interviewed)  and  one  half-time  person  of  the  MMS 
(202)  496-6193 

Type  of  Organization 

federal  agency,  database  producer 

Recipients  of  Instruction 

Users  or  prospective  users  of  MEDLINE  and  other  NLM  online  databases 
Class  Size(s) 

Introductory  called  "Initial  Training":  18  to  20  maximum 
Abbreviated  Initial  Training:  50 
Advanced  Training:  18  to  20  maximum 

Update:  20  (primarily  Technical  Resource  Persons  from  regional  offices) 
(Update  not  considered  training  by  NLM  staff) 

Location(s)  of  Instruction 

Initial  and  Advanced  Training:  NLM,  NLM* s  Regional  Medical  Libraries 
Abbreviated  InitiaL  Training:  NLM 
Updates:  NLM 

Duration(s)  of  Instruction 
Initial:  5  days 
Abbreviated  Initial:  3  days 
Advanced:  5  days 
Update:  1^  days 
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Frequency  of  Offering/Availability 

Initial:  monthly  at  NLM;  quarterly  at  an  RML 

Abbreviated  Initial:  4  per  year  planned  for  1981 

Advanced:  6  to  7  times  per  year  at  NLM,  6  to  7  times  at  an  RML 

Update:  annual 

Brief  Description  of  Nature  of  Instruction 

NLM  has  been  training  researchers  since  MEDLARS  (Medical  Literature 
Analysis  and  Retrieval  System)  was  first  automated  in  the  mid- 1960's  as  a 
batch  system.  An  initial  8-month  training  program  was  progressively  shortened 
to  6  and  then  to  2  months.  When  MEDLINE  (MEDLARS  On-Line)  was  activated  in 
1971,  the  training  program  was  redesigned  and  reduced  to  3  weeks.  A  change 
to  the  present  format  was  instituted  in  March  1977.  The  present  format 
consists  of: 

a.  Self-instruction  with  MEDLEARN,  a  CBI  program 

b.  Initial  Training,  1  week  lecture  +  workbook  +  hands-on  practice 

c.  Self-administered  test  after  6  or  more  months 

d.  Advanced  Training,  1  week  lecture  +  workbook  +  hands-on  practice 

e.  Updates 

Additionally,  NLM  presented  one  Abbreviated  Initial  Training  session  of  3  days 
in  1980  that  did  not  include  hands-on  practice.  Because  it  was  well  received, 

4  Abbreviated  Initial  sessions  are  scheduled  for  ]98], 

MEDLEARN  was  originally  developed  by  the  George  Washington  University 
Medical  Center  under  contract  to  NLM.  This  computer-based  instruction  (CBI) 
program  is  currently  being  modified  and  maintained  in-house.  Trainees 
receive  an  instruction  kit  and  a  special  password  for  free  use  of  MEDLEARN 
6  weeks  before  an  Initial  Training  course  begins.  They  are  expected  to  gain 
initial  familiarity  with  logging  on  and  off  and  with  rudimentary  aspects  of 
the  system  before  they  arrive.  Staff  of  the  MEDLARS  Management  Section  provide 
such  augmenting  assistance  as  may  be  needed  by  telephone.  Most  (but  not  all) 
of  the  fright  of  terminals  is  overcome  before  the  trainees  arrive  for  the 


Initial  Training  course 
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Organizations  that  wish  to  use  MEDLINE  must  send  at  least  one  person 
to  NLM  for  training.  The  organization  does  not  get  an  account  number  until 
one  person  has  received  training.  However,  because  of  class  size  limitations, 
only  one  person  from  an  organization  may  attend  a  course  at  any  one  time. 

Both  the  Initial  and  the  Advanced  Training  courses  are  held  in  a  class¬ 
room  in  the  new  Lister  Hill  National  Center  Building.  The  classroom  is  equipped 
with  10  TI  Silent  700  terminals;  there  are  no  CRTs  or  TV  monitors.  The 
instruction  methodology  consists  of  a  combination  of  lecture,  referral  to 
workbook  and  other  handouts,  and  use  of  the  terminal.  Two  students  are  assigned 
to  a  terminal.  The  workbooks  for  each  of  the  courses  is  detailed  and  voluminous 
(approximately  300  pages  each).  They  have  been  designed  to  encompass  the 
classroom  content  so  that  students  can  devote  their  attention  to  the  instruction 
rather  than  feel  compelled  to  take  copious  notes. 

Trainees  are  expected  to  gain  at  least  6  months  of  work  experience  before 
applying  for  the  Advanced  course.  Whey  they  apply,  they  receive  a  pretest. 

The  current  pretest  consists  of  10  pages  of  detailed  questions  about  use  of 
the  command  language  and  various  databases  plus  exercises  to  be  performed 
online  using  MEDLINE,  CANCERLIT/CANCERPROJ,  and  CHEMLINE.  The  pretest  responses 
are  brought  to  the  Advanced  course  and  discussed  on  the  first  day.  The  pretest 
serves  as  a  screen  to  preclude  attendees  without  the  requisite  knowledge. 

Many  different  types  of  organizations,  not  only  medical  schools,  hos¬ 
pitals,  and  research  institutions,  have  become  users  of  MEDLINE.  Over  1200 
unique  institutions  are  now  users  and  about  2000  passwords  have  been  issued. 
Because  of  the  heavy  demand  for  training,  NLM  tried  an  Abbreviated  Initial 
course  for  50  people  in  1980.  Of  3  days  duration,  it  is  held  in  an  auditorium- 
type  room  and  does  not  include  hands-on  practice.  The  acceptance  of  the  course 
by  attendees  has  led  NLM  to  schedule  4  for  1981. 


<• 
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NLM  issues  certificates  to  trainees,  and  trainees  receive  Continuing 

Education  (CE)  credits  from  the  Medical  Library  Assocation.  The  credit 

allocations  are: 

Initial  Training:  3.5  CE 
Advanced  Training:  3.5  CE 

Auditor  of  either  of  above  (no  hands-on  practice):  2.0  CE 

Abbreviated  Initial:  2.0  CE 

Abbreviated  Advanced:  2.0  CE  (see  below) 

Credit  is  given  only  for  courses  taught  by  NLM  staff  or  staff  authorized  by 
NLM.  At  present,  only  one  group  has  been  authorized  by  NLM  to  provide  train¬ 
ing,  namely  the  Regional  Medical  Library  at  the  University  of  California,  Los 
Angeles.  The  RML/UCLA  teaching  staff  received  its  training  at  NLM  and  taught 
at  NLM  under  the  observation  of  NLM  staff  prior  to  gaining  authorization. 

NLM  schedules  the  courses  to  be  taught  at  RML/UCLA,  selects  the  students, 
provides  the  instruction  materials,  forwards  the  certificates,  and  certifies 
the  students  to  the  MLA  after  training. 

Only  one  update  session  is  conducted  by  NLM  staff.  This  is  a  lk  day 
briefing  primarily  for  Technical  Resource  Persons  (TRPs)  from  the  eleven 
regions.  TRPs  are  selected  by  the  RMLs  and  may  be  either  members  of  RML 
staffs  or  from  an  educational  institution  in  the  region.  Updates  introduce 
TRPs  to  new  files,  file  designs,  changes  in  the  MESH  vocabulary,  and  anticipated 
system  modifications.  NLM  produces  printed  update  material  and  gives  150-200 
copies  to  each  TRP.  Each  TRP  is  expected  to  conduct  at  least  one  1-day 
update  in  his/her  region  annually  at  which  NLM' s  material  is  distributed. 

TRPs  and  the  RMLs  are  free  to  conduct  additional  training  courses  of  their 
choosing. 

NLM  courses  are  limited  to  NLM  system  users.  Some  NLM  databases  are 
available  from  Bibliographic  Retrieval  Services,  Inc.,  and  Lockheed  Informa¬ 
tion  Systems  announced  the  forthcoming  availability  of  some  databases. 

Although  in  actuality  many  BRS  users  are  also  NLM  users,  NLM  has  not  serviced 
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users  solely  with  the  BRS  system  and  does  not  plan  to  service  solely  Lockheed 
users. 

Improvements  Desired/Planned 

Success  of  the  Abbreviated  Initial  course,  described  above,  has  led  to 
a  decision  to  offer  this  course  quarterly  in  1981.  Additionally,  it  is  planned 
to  offer  an  Abbreviated  Advanced  course  at  least  once  in  1981  to  test  audience 
acceptability.  However,  presentation  of  the  Abbreviated  Initial  course 
indicated  the  desirability  of  including  online  demonstrations  observable  on 
TV  monitors  during  the  lecture  sessions.  It  is  planned  to  incorporate  this 
teaching  methodology  in  all  Abbreviated  courses  offered  in  1981. 

NLM  suspects  that,  with  the  diversity  of  user  organizations,  a  single 
Advanced  course  may  be  less  needed  than  a  series  of  advanced  one-day  workshops, 
each  centered  on  a  particular  subject  area.  An  Advanced  course  format  that 
groups  content  in  such  a  way  as  to  permit  attendees  to  select  topics  and  days 
most  appropriate  to  them  may  be  developed  and  offered  in  1981. 

Some  thought  has  been  given  to  grouping  trainees  to  create  more  homo¬ 
geneous  groups  and  thus  perhaps  improve  learning  for  all.  Some  trainees 
exhibit  "terminal  phobia;"  separate  classes  for  them  might  permit  a  faster 
teaching/ learning  rate  in  other  classes.  Trainees  from  different  user  organi¬ 
zations,  for  example,  hospital  libraries,  research  laboratories,  chemical 
companies,  have  different  needs  and  expectations.  It  is  thought  that  grouping 
by  type  of  user  organization  might  also  improve  teaching/ learning  effectiveness. 

A  redistribution  of  workload  in  the  MEDLARS  Management  Section  may 
occur  in  1981.  This  may  result  in  assigning  2  staff  members  full-time  teaching 
responsibility  and  giving  part-time  responsibility  to  1^  others.  At  present, 
each  person  in  the  MMS  is  responsible  for  correspondence  from  a  region  and 
each  takes  turns  at  the  user  service  desk. 

NLM  staff  are  not  expecting  to  give  courses  in  the  field  in  1981.  They 
have  a  very  limited  travel  budget,  and  they  are  not  permitted  to  accept  payment 
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from  user  organizations  to  defray  travel  costs.  Heretofore,  field  courses 
were  given  by  two  staff  members.  It  is  felt  that  this  causes  too  much  of  a 
mental  and  physical  drain  on  the  instructors.  For  courses  given  at  NLM, 
system  and  subject  specialists  are  brought  in  to  teach  their  specialties, 
thus  relieving  the  regular  instructors  from  continuous  teaching  for  intervals 
of  time. 

NLM  may  increase  the  number  of  organizations  authorized  to  teach  NLM 
courses.  The  RML  at  the  University  of  Nebraska  is  presently  under  considera¬ 
tion,  and  perhaps  one  or  two  additional  RMLs  will  receive  authorization. 

In  a  periodical  paper,  NLM  staff  announced  a  future  availability  of  a 
CBI  program  called  TOXLEARN.  Work  on  this  program  was  discontinued  because 
of  staff  shortages.  It  is  not  certain  whether  work  on  TOXLEARN  will  be  resumed 
in  1981. 

A  CBI  program  called  CUEMLEARN  has  been  written  and  is  now  being 
evaluated.  No  date  for  release  to  users  has  been  set, 

MEDLEARN  at  present  is  only  available  to  users  at  particular  times  and 
this  may  also  be  the  pattern  for  CHEMLEARN.  CHEMLEARN  probably  requires  more 
computer  capacity  than  MEDLEARN.  It  is  the  capacity  requirements  of  these 
CBI  programs  that  limit  their  availability.  However,  in  1980,  NLM  replaced 
its  IBM  370/158  with  a  370/168  and  at  the  close  of  1980,  new  storage  units 
were  installed.  This  may  facilitate  the  availability  of  the  CBI  programs. 

References 
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Laura  J.  Eisenberg  et  al.,  "MEDLEARN,  A  Computer-Assisted  Instruction  (CAI) 
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Type  of  Instruction 

Principal  mode  of  delivery:  Lecture 
Auxiliary  modes  of  delivery:  Slides, 

Workbooks, 

Hends-on  practice, 

Supplementary  printed  materials 

Organization  Providing  Instruction 

University  of  Dayton  Research  Institute  (UDRI) 

Kettering  Engineering  &  Research  Labs. 

Room  445 

Dayton,  Ohio  45469 
Instructors 

J.  Kirk  Brashear,  Project  Director 
Linda  M.  Swartzel 
(513)  229-3711 

Type  of  Organization 

public,  academic,  non-profit  institute,  contractor- supplied  instruction 
Recipients  of  Instruction 

Individuals  designated  by  the  contractor  (Foreign  Technology  Division, 
AFSC,  Wright-Patterson  AFB);  backgrounds  of  recipients  have  varied  from 
the  clerical  and  paraprof essional  to  librarian  to  end-user,  a  wide  range 
usually  present  in  each  class;  the  typical  student  is  a  non-information 
professional  in  the  GS-4  to  -7  range. 

Glass  Size(s) 

Introductory  course:  8  to  10 
Advanced  course:  6  to  8 
SDI  Profiles  course:  8  to  12 

Location  of  Instruction 

two  adjacent  rooms  at  the  UDRI 

Duration(s)  of  Instruction 

Introductory:  3  days  (Tuesday-Thursday) 

Advanced:  \\  days 
SDI  Profiles:  Ik  days 
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Frequency  of  Offering/Availability 
Introductory:  monthly 
Advanced:  semimonthly 
SDI  Profiles:  semimonthly 

The  Advanced  and  SDI  Profiles  are  offered  back-to-back,  the  Profiles 
course  immediately  following  the  Advanced  course. 

Background 

The  database  (actually  a  family  of  databases)  for  which  this  instruction 
program  provides  access  training  is  called  CIRC  II  (Central  Information  Ref¬ 
erence  and  Control  II).  It  is  the  information  system  of  the  DOD  Scientific 
and  Technical  Intelligence  Information  Support  Program  (STIISP).  The  STIISP 
meets  the  intelligence  information  needs  of  DOD' s  S&T  intelligence  production 
agencies,  namely  the  Naval  Intelligence  Support  Center,  the  Foreign  Science 
and  Technology  Center,  the  Medical  Intelligence  Information  Agency,  the  Missile 
Intelligence  Agency,  and  the  Foreign  Technology  Division  of  AFSC  where  CIRC  II 
is  housed. 

Each  of  the  five  intelligence  production  agencies  identifies  documents 

for  CIRC  II  and  provides  the  following  information  for  entry  into  the  database: 

Document  identification  and  attributes 
Geographic  content 
Temporal  content 

Personalities,  facilities,  nomenclature,  and  interrelationships 
Subject  content 

The  database  is  subdivided  into  an  unclassified  portion,  currently  avail¬ 
able  online  during  the  morning  (0830  to  1315  EDT,  M-F)  and  a  classified  (or 
secured)  portion,  currently  available  online  in  the  afternoon  (1400  to  1700  EDT, 
M-F).  The  database  is  subdivided  into  several  subsets.  Included  among  those 
available  mornings  are  the  following: 

PFLN  Files  (Personality,  Facility/Organization,  Location,  Nomenclature) 
Personality  -  a  file  of  surnames 

Facility/Organization  -  an  index  to  organizations,  facilities, 
military  bases  and  units,  and  post  box  numbers 
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Location  -  an  index  to  geographic  coordinates,  cities  and 
higher  geopolitics  ,  geophysical,  and  geographic  areas 
up  to  but  not  including  countries,  oceans  and  seas 
Nomenclature  -  an  index  for  materials,  system  and  hardware 
designations  and  code  names,  ship  names  and  classes, 
space  and  missile  event  numbers,  aircraft  registra¬ 
tion  numbers,  etc. 

UPDT  -  CIRC  II  records  for  documents  processed  during  the  current 
year  and  the  previous  year 
XLAX  -  index  to  S&T  translations 

CREF  -  cross  reference  list  of  CIRC  II  sequence  numbers,  original 
document  numbers,  and  microfiche  film  numbers 
PRQL  -  a  library  of  existing  SDI  profiles 

The  UDRI  initially  developed  and  provided  the  instruction  programs  under 
a  sole-source  procurement.  Since  1979,  it  has  conducted  18  Introductory 
courses,  4  Advanced  courses,  and  8  Profiles  courses.  In  the  Fall  of  1980, 
it  received  a  contract  renewal  under  an  FTD  RFP. 

Brief  Description  of  Nature  of  Instruction 

On  two  counts  --  the  complexity  of  the  databases  to  be  learned  and 
the  heterogeneous  mix  of  students  --  this  instruction  situation  differs 
appreciably  from  most  others.  To  promote  learning,  the  instructors  have  devised 
a  methodology  that  combines  the  extensive  use  of  slides  with  lectures  and 
extensive  terminal  practice. 

As  indicated  in  the  Background  section  above,  document  records  contain 
not  only  bibliographic  information  but  also  a  variety  of  other  descriptors 
used  by  the  intelligence  community  coupled  with  several  sets  of  numeric  codes. 

In  the  Introductory  course,  students  must  learn  not  only  an  access  language 
(a  modified  version  of  IBM's  STAIRS)  but  also  the  structures  of  several  data¬ 
bases.  A  measure  of  volume  as  well  as  complexity  may  be  suggested  by  noting 
that  the  PFLN  codes  for  a  single  symposium  or  handbook  type  of  document  could 


require  500  lines  of  printout 
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The  instructors  have  no  control  over  selection  of  the  individuals  who 
come  for  training.  This  selection  is  made  by  the  user  organizations,  63  in 
number,  that  consist  mainly  of  federal  intelligence  agencies  and  a  few  of  their 
contractors  (e.g.,  Battelle  Memorial  Institute).  For  some  di f f icult- to-under- 
stand  reason,  the  majority  of  persons  sent  by  the  agencies  are  non-technical 
and  non-professional.  This  may  have  some  bearing  on  the  turnover  that  apparently 
occurs  among  these  individuals,  because  UDRI  has  received  a  constant  flow  of 
about  80  people  per  year  for  the  Introductory  course  and  about  100  people  for 
the  Profiles  and  Advanced  courses.  Approximately  200  people  are  active  end- 
users  in  the  63  agencies  at  the  present  time.  The  instructors  see  the  need 
for  course  materials  and  course  content  that  can  promote  learning  despite  wide 
differences  in  background  and  experience  among  the  students. 

An  additional  complexity  is  the  existence  of  separate  classified  and 
unclassified  files.  As  specified  above,  files  containing  only  unclassified 
terms  are  accessible  in  the  mornings  and  classified  files  (having  confidential, 
secret,  and  top  secret  classifications)  are  accessible  in  the  afternoons. 

The  initial  thrust  in  the  Introductory  course  is  getting  students 
accustomed  to  working  at  terminals.  On  the  first  morning  after  administrative 
matters  are  finished,  students  are  seated  at  terminals  and  "stepped  through" 
logging  onto  the  system  and  some  simple  retrieval.  UDRI's  teaching  facility 
consists  of  a  classroom  with  audiovisual  equipment  and  an  adjacent  terminal 
room.  The  terminal  room  contains  4  Teletype  model  40/8' s  (keyboard,  CRT  and 
printer;  users  more  typically  have  model  40/2* s)  and  could  be  cramped  with 
10  to  12  people  in  it.  Because  of  the  limited  number  of  terminals,  individuals 
in  2  or  3  person  groups  take  turns  doing  and  watching. 

The  general  pattern  of  instruction  is  to  alternate  30-minute  sessions 
of  classroom  and  online  practice.  At  the.  first  terminal  session,  students  are 
told  the  keys  to  press.  Thereafter,  thcir'practice  sessions  pragmatically 
test  what  they  learned  in  the  classroom  session. 


-  37  - 


For  security  reasons,  FTD  monitors  the  online  usage  of  the  databases. 
Records  of  users'  search  strategies  and  online  printouts  have  been  given  to 
UDRI  for  analysis.  Some  general  patterns  of  search  behavior  are  discernible, 
but  an  inability  to  discuss  the  strategies  with  the  searchers  leaves  unanswered 
questions.  One  common  pattern  is  the  use  of  simple  search  strategies,  i.e. , 
the  failure  to  take  advantage  of  the  retrieval  power  of  codes  and  Boolean 
combinations.  This  may  result  from  the  inquiry  the  searcher  has  received  from 
the  client,  but  it  more  likely  results  from  the  searcher's  limited  experiential 
background  and/or  an  insufficient  knowledge  of  the  databases.  Another  common 
pattern  is  an  infrequent  use  of  HELP  messages.  Some  feedback  indicates  that 
searchers  prefer  to  seek  help  in  the  workbooks  UDRI  has  prepared  than  from 
online  aids.  Ms.  Swartzel  believes  the  system's  error  and  help  messages  may 
be  underused  because  they  are  in  language  that  is  not  readily  understood. 

A  significant  feature  of  the  Profiles  course  is  the  availability  of 
PROL,  a  database  of  current  SDI  profiles.  Searchers  are  encouraged  . o  use 
this  database  as  an  aid  in  profile  development.  A  separate  database,  PROM, 
exists  for  profile  construction  and  maintenance.  The  function  of  PROM  is 
similar  to  that  of  SDC's  ORBIT  database.  The  command  language  (and  computer 
program)  for  profile  development  was  written  by  BRS  (Bibliographic  Retrieval 
Services,  Inc.),  a  commercial  vendor  that  also  employs  a  modified  version  of 
STAIRS.  Parenthetically,  BRS  needed  9  months  to  debug  the  program  that  is 
now  being  maintained  by  FTD.  The  Profiles  class  assumes  a  knowledge  of  CIRC  II 
system  commands  and  strategy  formulation  and  is  limited  to  the  development 
and  maintenance  of  profiles. 

Improvements  Desired/Planned 

A  drawback  in  this  program  is  a  relationship  between  FTD  and  the  con¬ 
tractor  that  has  effectively  shut  the  contractor  off  from  feedback  from 
students.  FTD  has  instructed  users  to  call  in-house  FTD  monitors  who  have 
rarely  referred  callers  to  UDRI  instructors.  The  in-house  monitors  are  con- 


versant  with  administrative  and  security  matters  but  not  with  database  content 
and  search  strategy.  In  discussing  UDRI's  desire  to  help  users  with  content 
questions,  I  was  informed  by  FTD  staff  that  this  would  be  looked  into. 

Another  mechanism  for  contact  with  former  students  could  be  a  newsletter. 
FTD  has  been  issuing  a  quarterly  document  of  30  to  50  pages  that  consists  pri¬ 
marily  of  administrative  memoranda.  FTD  personnel  thought  of  this  document  as 
a  newsletter.  They  appeared  to  look  favorably  on  a  suggestion  of  a  newsletter 
similar  to  Lockheed's  Chronolog  that  UDRI  is  willing  to  produce. 

The  UDRI  instructors  have  given  thought  to  developing  two  types  of 
form  letters.  The  first  is  targeted  for  the  student's  employer  at  the  time 
the  student  is  designated  for  the  Introductory  course.  It  would  provide 
guidance  to  the  supervisor  on  the  support  the  student  needs  to  be  an  effective 
searcher.  The  second  letter  would  be  a  follow-up  to  the  student  a  short  time 
after  he/she  completed  the  course.  It  would  solicit  feedback  on  course 
effectiveness  and  on  the  search  activities  being  performed  on  the  job. 

Mr.  Brashear  has  been  thinking  about  the  advantages  that  a  static 
database  of  limited  size  might  have  for  training.  With  such  a  database,  the 
University  of  Pittsburgh  On-Line  Training  Center  (see  pages  39-41  )  is  able 
to  know  in  advance  the  retrievals  possible  with  search  queries  and  is  thus 
able  to  diagnose  students'  search  strategy  errors.  It  is  not  known  whether 
FTD  may  be  willing  to  create  a  training  file. 

References 
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Type  of  Instruction 

Principal  mode  of  delivery:  Hands-on  online  practice 
Auxiliary  modes  of  delivery:  Lecture, 

Printed  manuals, 

Supplementary  printed  handouts 

Organization  Providing  Instruction 

On-Line  Training  Center  (Pitt) 

University  of  Pittsburgh 
LIS  Building,  Room  808 
Pittsburgh,  PA.  15260 

Instructors 

Dr.  Elizabeth  E.  Duncan,  Coordinator 
Patricia  J.  Klingensmith 
Nina  M.  Ross 
(412)  624-5218 

Type  of  Organization 
public,  academic 

Recipients  of  Instruction 

Any  person  interested  in  computerized  searching;  no  restrictions 

Class  Size 

Not  to  exceed  10 

Location  of  Instruction 

Training  center  at  the  University 

Duration  of  Instruction 

3  days  (from  1  P.M.  Monday  to  noon  Thursday) 

Frequency  of  Of faring/Availability 

Approximately  one  session  monthly 

Brief  Description  of  Nature  of  Instruction 

This  education  program  emphasizes  learning  of  a  particular  search  system 
on  the  thesis  that  "a  thorough  knowledge  of  the  system  is  the  basis  for  effi¬ 
cient  and  intelligent  searching." 

At  present,  only  two  systems  arc  taught:  Lockheed's  DIALOG  and  SDC's 
ORBIT.  Only  one  system  is  taught  per  training  course.  The  course  for  each 
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system  is  available  in  alternate  months.  For  example,  in  January  1981, 

ORBIT  was  offered  and  in  February,  DIALOG. 

Although  the  lecture  mode  is  used  to  begin  each  course,  online  work  is 
begun  as  soon  as  possible.  Online  work  constitutes  about  50%  of  students’ 
activities  on  the  second  and  third  days.  For  this  reason,  instruction  is 
confined  to  the  On-Line  Canter  that  is  equipped  with  10  student  stations, 
each  having  a  keyboard-and-printer  terminal  (TI  Silent  700),  an  instructor's 
terminal  equipped  with  a  CRT,  and  three  21-inch  TV  monitors.  The  class  size 
is  limited  to  10  to  ensure  that  each  student  has  his/her  own  work  station. 

The  course  consists  of  introduction  to,  and  then  drill-and-practice 
on,  log  on  and  search  protocols.  Then  students  are  given  a  battery  of 
questions  (12  to  15)  to  search  in  a  particular  database.  Instructor  assist¬ 
ance  is  available  if  needed  during  the  period  that  students  develop  their 
search  strategies  and  execute  them.  Search  outcomes  and  pros  and  cons  of 
strategy  alternatives  are  discussed  in  primarily  a  lecture  mode.  This  is 
followed  by  further  instruction  in,  and  demonstration  of,  more  complex  aspects 
of  the  search  system  followed  by  another  battery  of  questions  for  students  to 
apply  to  a  different  database  followed  by  discussion  of  the  new  search  results. 

Students  are  given  a  workbook  for  the  search  system  that  includes  a 

[  glossary  and  detailed  explanations  and  illustrations  of  elements  of  the 

search  system.  They  are  also  given  content  descriptions  of  the  various  data- 

[ 

j  bases  that  are  mainly  reproductions  of  materials  issued  by  the  commercial 

f 

!  vendors.  The  most  difficult  level  of  learning  for  the  students  is  the  pro- 

I 

i  cessing  of  a  new  set  of  search  questions  in  databases  that  were  not  previously 

discussed  in  class  sessions.  This  requires  them  to  use  their  printed  materials 
and  is  a  test  of  their  ability  to  work  independently  as  they  would  be  required 


to  do  as  search  librarians 
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The  amount  of  online  time  that  is  used  in  this  instruction  program 
might  be  very  costly  except  for  a  computer  technique  employed  by  the  program 
designers.  DIALOG  and  ORBIT  emulators  have  been  written  for  a  PDF  VAX  located 
in  the  University’s  computer  center.  The  emulators  were  written  by  a  computer 
science  student  who  updates  them  as  features  of  the  DIALOG  and  ORBIT  systems 
are  added  or  changed.  Tapes  containing  information  on  a  small  number  of 
documents,  typically  1500  to  3000,  are  loaded  in  the  Pitt  computer  instead  of 
entire  databases.  Thus  teleprocessing  (e. g. ,  TYMNET,  TELENET)  and  computer 
costs  of  commercial  vendors  are  not  incurred.  A  total  cost  of  $10/hour  was 
estimated  for  use  of  the  VAX  system.  One  hour  of  student  use  of  the  commercial 
system  is  included  to  demonstrate  effects  of  complete  databases  and  timesharing 
with  other  users  on  computer  system  responsiveness  and  retrieval  magnitudes. 
Improvements  Desired/Planned 

A  larger  classroom  is  planned  with  a  capacity  for  20  to  24  terminal 
workstations.  A  larger  class  size  (15)  will  then  be  tried.  Dr.  Duncan  does 
not  want  to  lose  the  personalized  instruction  aspect  of  the  present  program 
that  would  occur  with  too  large  a  class. 

Little  change  is  seen  as  needed  in  the  instruction  materials.  The  one 
program  of  instruction  on  the  access  system  (DIALOG,  ORBIT)  is  considered 
sufficient  and  appropriate  for  this  Center.  In-depth  education  about  particular 
databases  and  about  the  most  effective  applications  of  the  access  systems  to 
particular  databases  is  considered  the  province  of  each  database  producer. 

No  advanced  course  on  access  systems  per  se  is  believed  necessary. 

References 

Elizabeth  E.  Duncan,  Patricia  J.  Klingensmith  &  Nina  M.  Ross,  "Manual  for 
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Type  of  Instruction 

Principal  mode  of  delivery:  Lecture  integrated  with  hands-on  practice 
Auxiliary  mode  of  delivery:  Manuals 

Organization  Providing  Instruction 

House  Information  Systems  (H.I.S.) 

Committee  on  House  Administration 

U.  S.  House  of  Representatives 

House  Annex  2  (2nd  &  D  Streets,  S.  W. ) 

Washington,  D.  C.  20515 

Instructors 

V,  Douglas  Hines,  Head,  Education  and  Training  Systems,  directs  an 
instruction  staff  of  four 

(202)  225-0217 

Recipients  of  Instruction 

Members  of  Congressmen's  staffs  (mainly  legislative  assistants  or  LAs) 
and  House  Committee  staffs  (about  1000  people  per  year) 

Class  Size 

6  to  8  maximum 

Location(s)  of  Instruction 

Training  room  in  House  Annex  2 

Training  room  in  Longworth  Building  (House  office  bldg.) 

Duration(s)  of  Instruction 

LEGIS,  SCORPIO:  two  2-hour  sessions 
JURIS:  one  2-hour  session 
Advanced  LEGIS:  one  2-hour  session 
SOPAD,  FAPRS,  MBIS:  one  1%  hour  session 

Frequency  of  Of f ering/Availability 
Dependent  on  demand 

Up  to  approximately  12  classes  weekly  is  within  planned  workload 
Background 

House  Information  Systems  (H.I.S.)  was  created  in  1971  and  provides 
computer  support  to  the  House  of  Representatives.  H.I.S.  has  grouped  its 
services  that  furnish  access  to  online  databases  under  the  name  MIN  -  Member 
Information  Network,  Currently,  6  databases  are  available: 


-L  -  - - -  ,  _  . 
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LEGIS  (Legislative  Information  and  Status  System) 

Contains  status  information  on  legislation  from  the  93rd 
Congress  to  the  present 

SCORPIO  (Subject-Content-Oriented  Retriever  for  Processing 
Information  Online) 

System  contains  7  databases;  see  description  of  CRS's 
SCORPIO  training  at  pages  51-56 

JURIS  (Justice  Retrieval  and  Inquiry  System) 

Contains  opinions  of  the  Supreme  Court,  Courts  of  Appeals, 
other  Federal  Courts,  the  U.S.  Code,  other  federal  regulations 

SOPAD  (Summary  of  Proceedings  and  Debates) 

Contains  information  about  House  floor  proceedings,  floor 
schedules,  and  summaries  of  vote  actions 

FAPRS  (Federal  Assistance  Program  Retrieval  System) 

Is  the  online  version  of  the  Catalog  of  Federal  Domestic 
Assistance,  contains  federal  loan  and  grant  program  data 

MBIS  (Member  Budget  Information  System) 

Contains  budget  tables  prepared  and  updated  by  the  House 
Budget  Committee  including  data  by  program  groupings, 
action  on  appropriation  bills,  aggregates  by  spending 
committees,  definitions  of  terms,  and  early  warning  reports 

H.I.S.  produces  the  LEGIS,  SOPAD  and  MBIS  databases.  The  Congressional 
Research  Service,  Library  of  Congress,  produces  SCORPIO.  The  Department  of 
Justice  produces  JURIS.  The  Office  of  Management  and  Budget  has  the  responsi¬ 
bility  for  maintaining  FAPRS. 

Other  activities  of  H.I.S.  include  development  and  maintenance  of  an 
electronic  voting  vstem  (in  continuous  operation  for  over  two  years),  assess¬ 
ments  of  office  automation  needs  and  aid  with  acquisition  and  installation  of 
computer  equipment  and  services,  provision  and  maintenance  of  text  processing 
software,  provision  and  maintenance  of  a  conversational  graphics  system  called 
IMAGE,  and  operation  of  a  central  computer  facility  for  the  House. 
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Brief  Description  of  Nature  of  Instruction 

A  distinguishing  feature  of  the  H.I.S.  instructional  approach  is  the 
emphasis  accorded  to  hands-on  practice  by  the  student  in  contrast  with 
instructor-performed  online  demonstrations  or  lesser  amounts  of  online  practice 
common  in  other  methodologies.  Other  differences  are:  (a)  the  short  training 
times  considered  sufficient  and  (b)  the  simple  structure  of  some  databases 
(SOPAD,  FAPRS,  MBIS). 

For  two  of  the  databases  having  complex  content  structures,  SCORPIO  and 
JURIS,  instruction  programs  have  been  developed  by  their  federal  agency  pro¬ 
ducers.  The  instruction  programs  of  the  Congressional  Research  Service  and 
the  Department  of  Justice  are  available  to  the  same  user  groups  as  those  of 

H.I.S. 

The  content  of  the  H.I.S.  training  manuals  suggests  that  individuals 
are  being  provided  with  bare  essentials  to  enable  them  to  perform  a  few 
frequently  occurring  types  of  retrieval  as  the  need  for  information  arises. 

Most  of  the  trainees  are  not  expected  to  be  searchers.  Many  of  the  trainees 
may  access  the  databases  only  occasionally.  Trainees  are  apparently  forced 
by  the  pressure  of  their  work  to  cancel  even  the  short  scheduled  sessions 
with  some  frequency,  a  further  indication  of  their  relationship  to  searching 
vs.  other  activities.  Additionally,  users  may  obtain  search  assistance  at 
any  time  from  the  H.I.S.  User  Assistance  Office  (UAO)  that  has  a  staff  of  six 
people  who  work  in  shifts  to  ensure  that  assistance  is  available  at  any  hour 
the  House  is  in  session  (including  early  morning  hours).  UAO  staff  go  to 
Congressmen's  offices  to  provide  assistance  when  telephone  help  doesn't  suffice. 
Persons  trained  in  the  systems  may  request  retraining  as  they  deem  it  necessary 
to  refresh  their  knowledge. 

All  of  the  four  instructors  are  competent  to  teach  the  use  of  all  of 
the  databases.  Each  may  be  assigned  up  to  8  hours  of  teaching  per  week. 
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Each  course  is  taught  by  a  single  instructor.  The  instructors  have  found  that 
trainees  learn  faster  when  two  share  a  terminal.  The  Hazeltine  1510,  A  CRT 
and  keyboard,  is  the  terminal  in  current  use.  The  learning  stations  do  not 
have  printers  by  trainee  preference.  It  is  thought  that  they  may  not  wish  to 
have  hard-copy  records  of  their  learning  errors.  Although  many  Congressmen's 
and  committees'  office  terminals  are  equipped  with  printers,  it  is  believed 
that  many  of  this  user  group  query  files  only  to  obtain  one  or  two  items  of 
information;  they  don't  need  a  record  of  the  response.  Trainees  are  instructed 
in  use  of  the  print  option  and  they  may  request  a  printout  of  their  terminal 
use  that  is  done  at  a  printer  in  another  location. 

Despite  notable  uses  of  information  technologies  by  the  House,  many 
of  the  staff  are  slow  to  adopt  the  automated  mode  of  searching  for  informa¬ 
tion.  H.I.S.  has  used  the  education  channel  6  of  the  House's  cable  TV  system 
to  advertise  the  availability  of  training  for  LEGIS,  its  most  heavily  used 
database.  The  videotape  prepared  by’ H.I.S.  is  broadcast  twice  weekly,  but 
the  response  from  the  TV  promotion  suggests  that  few  of  the  target  audience 
are  watching  (or  are  interested). 

Improvements  Desired/Planned 

At  present,  the  User  Assistance  Office  is  reactive,  responding  to  requests 
for  help.  Thought  is  being  given  to  monitoring  the  usage  of  databases  by 
persons  given  training  and  to  make  follow-up  inquiries  at  least  of  users  in 
the  lowest  quartile  to  gain  an  understanding  of  reasons  for  low  usage. 

H.I.S.  is  readying  a  new  version  of  its  printed  materials  for  its 
users.  Heretofore,  printed  materials  have  been  distributed  as  separates. 

In  1981,  a  looseleaf  binder  containing  all  of  the  manuals  and  quick-reference 
cards  will  be  made  available. 

Individuals  in  Congressmen's  state  offices  have  access  to  the  H.I.S. 


databases  although  not  much  remote  use  of  them  has  been  made.  A  trend,  however, 
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is  being  discerned  of  greater  use  by  Congressmen  of  home  office  siaffs.  In 
anticipation  of  requests  for  training  at  locations  other  than  the  two  class¬ 
rooms,  is  examining  mechanisms  for  training  at  a  distance.  These  mechanisms 

may  also  be  useful  to  people  whose  schedules  do  not  permit  attendance  at  the 
live  training  sessions. 

One  methodology  is  the  use  of  audio  tapes  as  replacements  for  classroom 
instructors.  Two  tapes,  one  for  LEGIS  and  one  for  JURIS,  have  been  produced. 

They  are  to  be  evaluated;  they  have  not  yet  been  distributed.  The  sound  portion 
contains,  in  essence,  the  same  information  that  an  instructor  is  likely  to 
include  in  a  lesson  plan.  The  trainee  is  expected  to  be  at  a  terminal.  The 
tape  instructs  the  trainee  in  the  use  of  the  terminal  and  the  database  manual. 

The  trainee  is  "walked"  through  elementary  features  of  the  command  language 
and  the  database  and  is  continuously  given  exercises  and  searches  to  perform 
at  the  terminal  followed  by  verbal  descriptions  of  what  he/ she  should  have 
done. 

A  second  methodology  is  computer-based  instruction  (CBI).  H.I.S. 
staff  explored  three  CBI  software  systems  --  PLATO  of  Control  Data  Corp. , 
SCHOLAR/TEACH  of  Boeing  Computer  Services,  and  IIS  (Interactive  Instructional 
System)  of  IBM.  SCHOLAR/TEACH  3,  Version  4  is  currently  being  used  to  write 
a  CBI  program  that  teaches  the  legislative  process.  This  program,  when  finished, 
will  not  replace  any  of  the  online  training.  It  could,  however,  be  a  precursor 
training  module  for  new  Congressional  staff.  Knowledge  of  the  legislative 
process  is  assumed  in  training  for  LEGIS,  SCORPIO,  and  other  databases. 

Other  uses  of  CBI  are  being  sought  that  will  complement  rather  than  replace 
current  instruction  in  the  use  of  online  databases. 
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Type  of  Instruction 

Principal  mode  of  delivery:  Self-instruction  online 

(a  form  of  computer-aided  instruction) 

Auxiliary  modes  of  delivery:  Lecture, 

Supplementary  printed  materials 

Organization  Providing  Instruction 

Principally  library  schools,  but  need  not  be  limited  to  academic  institutions 
in  the  near  future,  for  GAS  materials,  probably  chemistry  departments 

Developers  of  Instruction  Materials 

Lockheed  Information  Retrieval  Service  (LIRS) 

3460  Hillview  Avenue 
Palo  Alto,  CA.  94304 

Chemical  Abstracts  Service  (CAS) 

P.  0.  Box  3012 
Columbus,  Ohio  43210 

Contact  Persons 

For  LIRS:  Sophie  K.  Hudnut,  DIALOG  Training  Coordinator 

(415)  858-3810;  (800)  227-1927  (except  CA);  (800)  982-5838  (in  CA) 

For  CAS:  Diane  Yoder,  Manager,  User  Education 
(614)  421-6940  x2945 

Type  of  Organization 

LIRS:  commercial  online  services  vendor 
CAS:  not-for-profit,  database  producer 

Recipients  of  Instruction 

LIRS:  individuals  with  a  rudimentary  knowledge  of  DIALOG,  although  more 
skilled  persons  may  find  this  a  useful  refresher 
CAS:  individuals  needing  in-depth  knowledge  of  how  to  use  CA  databases 

Class  Size 

primarily  the  individual  learner 
Location  of  Instruction 

any  terminal  with  access  to  the  Lockheed  computer  system 
Duration  of  Instruction 


varies,  determined  by  the  individual’s  learning  needs  and  preferences 
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Frequency  of  Offering/Availability 
an  the  learner's  option 

Brief  Description  of  Nature  o£  Instruction 

This  mode  of  instruction  has  been  named  ONTAP  by  Lockheed,  its  principal 
developer;  this  is  an  acronym  for  ONline  Training  And  Practice. 

Currently,  there  are  three  ONTAP  files  in  the  Lockheed  system: 

File  201,  ONTAP  ERIC 
File  204,  ONTAP  CA  SEARCH 
File  231,  ONTAP  CHEMNAME 

Lockheed  has  plans  to  provide  ONTAP  files  for  each  of  the  databases  in  its 
system,  but  implementation  of  these  plans  does  not  seem  to  have  a  top  priority. 

ONTAP  files  are  small  samples  of  the  respective  databases.  By  working 
with  a  file  of  known  content  and  manageable  size,  instructors  can  formulate 
exercises  and  search  questions  that  they  can  apply  to  the  file,  examine  the 
response,  and  modify  to  fit  aspects  of  search  strategy  and/or  file  content  they 
want  to  demonstrate  to  students.  Then  they  can  assign  these  exercises  and 
questions  to  students  and  be  able  to  judge  from  students'  responses  the  level 
of  student  learning  and  expertise.  Alternatively,  questions  and  answers  can 
be  provided  in  a  workbook  usable  in  a  self-instructional  mode.  With  large 
and/or  open-ended  files,  instructors  either  couldn't  have  complete  foreknowledge 
of  the  response  or  lists  of  citations  would  be  too  lengthy  for  practical 
inclusion  in  a  workbook. 

Lockheed  produces  the  tutorial  printed  material  for  ONTAP  ERIC  and  the 
Chemical  Abstracts  Service  issued  a  250-page  manual  for  Files  204  and  231  in 
September  1980. 

ONTAP  ERIC  is  composed  of  the  1975  records  of  the  ERIC  database.  It 
contains  32,120  citations  that  appeared  in  the  1975  issues  of  Resources  in 
Educat ion  and  Current  Index  to  Journals  in  Education.  Lockheed  obtained  the 
cooperation  of  15  professional  librarians  in  user  organizations  to  evaluate 
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29  test  questions  and  search  responses.  The  questions  are  divided  into  three 
levels  of  complexity:  simple,  medium,  and  difficult. 

Students  are  given  a  description  of  ONTAP  ERIC  that  includes  the  list 
of  the  test  questions  and  instructions  for  using  the  database  in  an  online 
self-instructional  mode.  The  description  also  contains  references  to  other 
printed  materials  about  the  database.  The  self-instructional  program  is  stored 
in  Lockheed's  computer.  The  student  selects  a  question,  enters  a  search  strategy, 
and  obtains  a  result.  He/she  can  then  compare  his/her  result  with  the  response 
developed  by  the  professional  panel.  The  comparison  can  be  by  numeric  count 
(Number  of  citations  in  common)  and  by  citation  record  for  references  not 
retrieved  by  the  student  searcher.  By  diagnostically  examining  the  latter 
records,  the  student  is  supposed  to  be  able  to  identify  deficiencies  in  his/her 
search  strategy.  All  DIALOG  system  features  can  be  used  in  ONTAP  ERIC  except 
SEARCH-SAVE  and  PRINT  (the  offline  print  command;  the  online  print  command  is 
TYPE). 

The  CAS  files  consist  of  15,700  records  that  were  published  in  Issues 
15  and  16  of  Chemical  Abstracts  Volume  87  (two  weeks  of  October  1977).  In  a 
test  of  the  CAS  manual,  two  chemistry  students  at  Otterbein  College  and  Roches¬ 
ter  University,  respectively,  were  taught  the  logon  procedure  and  then  worked 
independently  with  the  manual  and  CAS  search  aids  for  two  weeks.  The  manual 
covers  fundamental  concepts  of  online  searching,  comparisons  of  manual  and 
computerized  searching.  Boolean  operators,  the  command  language,  the  structure 
and  content  of  the  CA  databases,  indexing  policies  and  practices,  and  search 
strategy  development.  The  largest  portion  of  the  manual  gives  suggested 
search  strategies  for  about  12  practice  problems.  The  CAS  tutorial  does  not 
include  the  relevance/recall  feature  in  ONTAP  ERIC. 

This  is  not  computer-assisted  instruction  in  the  conventional  use  of 
the  term,  but  the  ONTAP  concept  is  certainly  classifiable  as  a  form  of 
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computer-aided  instruction  in  that  computer  processing  is  an  integral  part  oi 
the  teaching/learning  process. 

LIRS  charges  $15  per  connect-hour  for  use  of  the  ONTAP  files.  CAS  does 
not  levy  a  royalty  fee  on  its  ONTAP  files.  LIRS  gives  2  hours  of  free  practice 
on  ONTAP  files  to  those  who  enroll  in  its  introductory  course,  the  System 
Seminar  (see  page  13). 

References 

"ONTAP  ERIC,"  File  201,  Preliminary  Edition,  June  1979.  Lockheed  DIALOG  Infor¬ 
mation  Retrieval  Service,  Palo  Alto,  CA. 

"CA  Search  for  Beginners:  An  Introduction  to  On-line  Access  to  CA  SEARCH," 
Lockheed  Version.  Chemical  Abstracts  Service,  Columbus,  Ohio,  1980. 

Harry  J.  Hamilton,  Erica  Scurr  &  Diane  K.  Yoder,  "An  Introductory  Manual  for 
Training  Science  Students  in  On-Line  Access  to  CA  Search,"  unpublished  paper 
presented  at  ASIS  Annual  Meeting,  Anaheim,  CA,  October  1980. 
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Type  of  Instruction 

Principal  mode  of  delivery:  Self-instruction  online 

Computer-based  instruction  (CBI) 

Auxiliary  modes  of  delivery:  Video  tape, 

Printed  materials. 

Demonstration  searches 

Organization  Providing  Instruction 

Information  Systems  Section  (CRS) 

Office  of  Automated  Information  Services 
Congressional  Research  Service 
Library  of  Congress 
Madison  Building,  Room  LM  222 
Washington,  D.  C.  20540 

Instructor 

Jeffrey  Griffith 
(202)  287-6447 

Type  of  Organization 

public,  federal  government,  database  producer 
Recipients  of  Instruction 

Congressional  staff  members,  Congressional  Research  Service  analysts 
Class  Size 

Pre-instruction  session:  3  to  12  people 
Instruction:  1  person  per  terminal 

Location  of  Instruction 

Terminal  room  in  ISS  office 

Duration  of  Instruction 
Learner  dependent 

Frequency  of  Of fering/Availability 

Learner  dependent  within  system  scheduling  possibilities 

Background 

This  instruction  program  trains  in  the  use  of  the  SCORPIO  system 
(Subject-Content-Oriented  Retriever  for  Processing  Information  Online).  The 


system  currently  contains  the  following  databases: 
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Congressional  Record  (CR)  Files 

The  CRs  for  the  94th  (2nd  Session),  95th,  and  96th  Congresses 
(1976,  1977-1978,  1979-1980  respectively)  that  contain  abstracts 
of  CR  pages  with  floor  debates  indexed  by  bill  numbers,  dates, 
names  of  members  of  Congress,  and  subject  terms.  These  are 
searchable  elements.  The  databases  are  produced  by  Capitol 
Services,  Inc. 

Bill  Digest  Files 

Data  from  the  publication  "Digest  of  General  Public  Bills  and 
Resolutions"  for  the  94th,  95th,  and  96th  Congresses.  These 
files  permit  a  bill's  passage  through  Congress  to  be  monitored. 
In  addition  to  an  abstract,  bill  digest,  and  revised  digest, 
the  files  contain  information  on  bill  sponsor  and  cosponsors, 
official  title,  short  title,  committees  of  origin,  referral, 
and  reporting,  floor  actions,  companion  bills,  and  public  law 
numbers. 

Major  Issues  File 

Titles,  subject  descriptors,  and  numbers  for  over  300  papers 
called  "Issue  Briefs"  written  and  kept  updated  by  CRS  staff 
members  on  major  public  policy  issues.  Each  Brief  contains 
a  definition  of  the  issue,  background  and  policy  analysis  in¬ 
formation,  references  to  Congressional  actions,  and  a  biblio¬ 
graphy. 

Citation  File 

Abstracts  and  bibliographic  data  for  periodical  articles, 
pamphlets,  GPO  publications,  UN  documents,  lobby-group  documents, 
and  other  publications  selected  by  CRS  staff  for  their  relevance 
to  major  public  policy  issues.  Searchable  elements  include 
LIV  (Legislative  Indexing  Vocabulary)  subject  terms. 

LC  Computerized  Catalog 

Bibliographic  data  extracted  from  the  MARC  database  for  books 
in  the  English  language  received  in  LC  since  1969,  books  in 
French  received  since  1973,  and  books  in  all  Roman  alphabets 
received  since  1977. 


National  Referral  Center  Resources 

Name,  address,  telephone  number,  topics  of  concern,  collection 
and  database  holdings,  publications,  and  dissemination  services 
for  organizations  willing  and  qualified  to  provide  information 
on  physical  science,  technology,  and  social  science  topics.  The 
file  is  maintained  by  the  National  Referral  Center. 

General  Accounting  Office  Recurring  Reports 

Abstracts  and  data  on  agency  names,  Congressional  recipient, 
mandatory  legislation,  oversight  committees,  frequency,  date, 
and  subject  content  for  all  recurring  reports  required  of  federal 
agencies  by  law.  GAO  produces  the  database. 

GAO  Information  Systems  and  Sources 

Abstracts  and  data  on  agency  name,  contact  point,  Congressional 
responsibility,  funding  category,  and  subject  content  for  infor¬ 
mational  documents  ( e. g. ,  annual  reports)  and  information  systems 
(e.g. ,  bibliographic  systems,  management  information  systems) 
produced  or  maintained  by  federal  agencies.  GAO  produces  the 
database. 

GAO  Program  Evaluations 

Abstracts  and  data  on  the  sponsoring  agency,  managing  agency, 
program  name.  Congressional  recipient,  public  availability,  and 
‘subject  content  for  about  1700  federal  programs  that  produce 
evaluations  of  their  activities.  GAO  produces  the  database. 

Brief  Description  of  Nature  of  Instruction 

Except  for  a  brief  non-CBI  beginning,  this  instruction  program  relies 
entirely  on  computer-based  instruction  (with  human  assistance  if  requested) 
to  train  students  in  the  use  of  the  first  four  databases  listed  above, 
namely  the  Congressional  Record,  Bill  Digest,  Major  Issues,  and  Citation 
files.  The  program  uses  the  PLATO  system  marketed  by  the  Control  Data  Corp. 

Trainees  are  usually  given  a  45-minute  briefing  in  groups  of  3  to  12. 
The  briefing  includes  a  14-minute  TV  film  in  which  Jeffrey  Griffith  presents 
introductory  information  about  the  four  databases,  shows  some  of  the  source 
material  about  which  information  is  extracted  and  stored,  and  mentions  the 
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need  to  learn  search  strategies  to  profitably  use  the  databases.  Printed 
material  about  SCORPIO  is  distributed  including  a  Survey  of  Interests  form. 

This  form  enables  the  trainee  to  know  in  advance  the  learning  options  available 
to  him/her  and  to  reach  a  decision  on  topics  for  which  instruction  is  desired. 

The  CBI  program  is  menu  driven,  and  the  Survey  of  Interests  material 
reappears  on  the  CRT,  forcing  a  selection  if  the  trainee  is  to  proceed.  A 
short  routine,  "Introduction  to  SCORPIO  Commands"  that  ends  with  7  test  questions 
that  must  be  answered,  precedes  learning  modules  about  the  databases.  This 
ensures  a  basic  knowledge  of  the  command  language  that  is  needed  for  database 
manipulation. 

The  CBI  program  was  written  by  Control  Data  Corp.  staff  under  contract 
to  CRS.  A  first  draft  was  written  between  January  and  March  1980  and  was 
tested  in  June  and  July.  Mr.  Griffith  produced  a  program  revision  in  August 
to  reduce  the  run  time.  (The  first  draft  also  rambled,  had  too  little  student 
involvement,  and  was  boring  in  my  judgment  based  on  a  small  portion  of  it 
that  I  worked  with.)  CDC  staff  produced  the  present  version  in  September 
and  October  1980. 

I  examined  the  program  primarily  via  the  modules  of  the  Citation  file. 

I  used  approximately  1\  hours  for  7  modules.  Satisfying  features  of  the 
program  include: 

a.  The  appearance  of  check  marks  beside  the  options  in  a  menu  that 
are  selected  so  that  selections  don't  have  to  be  remembered. 

b.  Reinforcements  through  restatements  appearing  on  the  screen  after 
test  questions  are  answered  correctly  to  insure  that,  should  the 
answers  have  been  guessed  at,  correct  learning  can  be  enhanced. 

c.  Highlighting  of  data  elements  of  a  record  by  enclosing  each,  in 
turn,  in  a  box  that  initially  flashes  during  an  explanation  of 
the  data  element.  This  has  the  effect  of  helping  the  trainee 
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locate  the  placement  of  the  data  element  in  relation  to  other  data 
elements  as  well  as  seeing  how  it  appears  in  a  record  (i.e.,  whether 
abbreviated,  with  words  or  numbers,  etc.). 

d.  Brevity  of  instructional  messages  with  many  requirements  for  student 
response.  This  contrasts  with  the  first  version  that  showed  long 
paragraphs  of  text,  a  characteristic  L  also  observed  in  several  PLATO 
programs  for  other  subject  matter.  Most  of  the  instruction  is  now 
packaged  in  1  to  3  sentence  sequences  that  seem  about  right  for 
attention  and  learning. 

e.  Frequent  insertion  of  questions  either  requiring  the  trainee  to  use 
the  keyboard  or  touch  an  area  of  the  screen.  The  change  from  one  to 
the  other  provides  a  useful  diversity, 

A  frustrating  feature  was  a  limitation  in  the  assistance  available  in 
HELP  routines.  I  tried  HELP  for  an  explanation  of  RETRIEVE.  The  present  pro¬ 
gram  gives  general  messages  and  requires  the  trainee  to  progress  through  a 
fixed  set  of  messages  that  supposedly  will  include  desired  information.  In 
one  use  of  HELP,  I  never  found  an  explanation  for  an  error  I  made  and  even 
though  I  could  proceed  through  the  instruction  program,  this  created  a  sense 
of  dissatisfaction. 

CRS  usually  schedules  1  or  2  hours  per  terminal  session  for  a  trainee 
rather  than  longer  periods  because  of  the  fatigue  factor.  I  experienced  a 
fatigue  point  after  55  minutes  and  left  the  terminal  after  2\  hours  because 
of  mental  and  physical  fatigue. 

Several  practice  problems  are  given  to  the  trainee  after  he/she  completes 
the  modules  selected  for  study.  These  may  be  worked  online.  They  give  the 
student  a  more  thorough  yardstick  for  measuring  the  amount  of  learning  that 
has  occurred  than  mere  response  to  the  program's  questions  does. 

The  non-CBI  beginning  is  a  14-minute  color  TV  program  that  features  a 
cast  of  one,  Mr.  Griffith.  The  film  holds  the  interest  of  the  viewer  because 
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it  is  not  merely  a  replica  of  a  lecture.  Through  verbal  description  and  visual 
images,  the  film  introduces  the  viewer  to  the  SOORPIO  system  and  the  content 
scope  of  the  online  databases.  It  gives  the  viewer  a  cursory  foundation  for 
choosing  appropriate  CBI  programs  for  detailed  study.  Mr.  Griffith  and  another 
staff  person  spent  a  combined  total  of  about  one  week  on  the  script.  The  staff 
person  worked  about  half  time  for  an  additional  7  to  8  week  period  on  the 
storyboard,  assembling  materials,  and  directing  and  editing  the  film.  The 
film  was  shot  in  two  days  in  a  GAO  studio. 

Improvements  Desired/Planned 

Several  changes  and  additions  were  made  to  the  command  language  in  1978 
and  1979.  An  adjacency  capability  will  probably  be  added  in  1981.  Several 
additions  may  be  introduced  into  the  CBI  program  to  reflect  the  97th  Congress 
(1981-1982).  No  other  major  system  changes  are  anticipated  in  the  near  future. 
Minor  modifications  are  expected  to  be  provided  to  trainees  by  means  of 
printed  materials. 
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FINDINGS,  PART  II  -  INSTRUCTION  PROGRAM  COMPARISONS 

Table  I  presents  a  comparison  of  the  instruction  programs  in  four 
major  areas:  (see  pages  68-75) 

1.  Purposive  Documentation 

goals,  objectives,  criteria 

2.  Education  Program 

components,  duration(s),  availability,  location(s),  participants 

3.  Resources 

human,  equipment,  facilities,  instructional  materials 

4.  Evaluation/Feedback  Mechanisms 

instruments,  tests,  follow-ups,  user  groups 

Since  little  has  been  published  on  the  evaluative  analysis  of  education 

programs  for  information  professionals,  some  comment  about  the  elements  of 

Table  I  may  be  helpful.  In  a  paper  written  by  the  investigator  several  years 

2 

ago,  characteristics  of  evaluation  methodologies  were  discussed.  In  the 
terminology  of  that  paper,  the  elements  of  Table  1  are  criterion  variables. 

An  education  program  designer  could  select  appropriate  variables  and  impose 
qualitative  or  quantitative  desiderata  on  them,  for  example,  a  student /instructor 
ratio  in  the  range  of  10/1  to  15/1.  These  specifications  could  then  serve  as 
program  design  guidelines  as  well  as  variables  by  which  programs  could  be 
periodically  evaluated.  The  elements  of  Table  I  were  developed  for  online 
instruction  programs.  Different  elements  may  be  needed  for  other  types  of 
courses. 

1.  PURPOSIVE  DOCUMENTATION 

Most  programs  are  weak  in  purposive  documentation.  Typically,  only 
very  general  overall  goals  are  stated.  Only  GLSIM  and  NLM  have  produced  at 
least  partially  measurable  objectives.  GSLIM  course  syllabi  are  required  to 
include  specific  objectives  and  exit  skills  students  are  expected  to  acquire. 

This  documentation  is  a  typical  requirement  in  academic  institutions.  NLM' s 


documentation  is  more  extensive;  objectives  have  been  developed  for  each  lec¬ 
ture.  Most  of  the  objectives  can  be  performance  tested.  NLM' s  "System  Mechanics" 
lecture,  for  example,  has  5  objectives,  one  of  which  is  "Describe  the  purpose, 
entry  format,  and  results  of  the  use  of  the  following  ELHILL  commands:  TIME, 

CAPS,  ...etc.  "  (28  commands  are  listed).  The  education  community  has  been 
promoting  the  use  of  measurable  objectives  as  an  aid  to  program  design  and 
evaluation.  It  can  be  contended  that  courses  without  prespecified  directions 
may  be  less  effective  in  conveying  need-to-know  information.  It  is  certainly 
true  that  the  absence  of  objectives  precludes  a  measure  of  accomplishment 
(unless  this  is  created  after  the  fact). 

2.  EDUCATION  PROGRAM 

Four  of  the  programs  have  a  multi-course  structure;  the  remainder  are 

either  single  course  programs  (GSLIM,  Pitt)  or  they  have  a  single  course  per 

database  (H.I.S.,  ONTAP,  CRS).  Each  of  the  four  has,  minimally,  an  introductory 

course  and  one  advanced  course.  LIRS  is  the  biggest  course  producer  with  12 

subject  seminars  in  such  fields  as  Biosciences,  Business,  Government  Documents, 

and  Nonbibliographic  Databases,  an  advanced  search  strategy  seminar,  a  system 

refresher  seminar,  and  periodic  system  updates  in  addition  to  its  introductory 

System  Seminar.  It  is  likely  that  LIRS  will  introduce  2  or  3  more  subject 

seminars  in  1981.  NLM  has  experimented  more  with  course  content,  duration, 

3 

and  format  than  any  of  the  other  organizations. 

In  duration,  all  short  courses  differ  markedly  from  for-credit  academic 
courses.  None  exceeds  a  one-week  period.  NLM,  that  had  an  8-month  training 
program  when  MEDLARS  (Medical  Literature  Analysis  and  Retrieval  System)  was 
first  automated  in  the  mid-19601 s  redesigned  to  a  3-week  program  for  MEDLINE*  s 
debut  in  1971  and  changed  to  its  present  multi-course  format  in  1977.  NLM  is 
now  exploring  modifications  of  its  5-day  advanced  course  to  shorter,  specialized 
segments  so  that  people  will  be  able  to  attend  only  those  portions  of  interest 
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to  them.  To  some  extent,  a  5-day  course  of  35  to  40  hours  provides  60-70%  of 
the  instructional  contact  time  of  an  academic  course.  Whether  concentration 
on  a  topic  for  a  shorter  period  or  internalization  of  the  topic  over  a  longer 
period  produces  greater  learning  and  retention  is  a  debatable  issue.  Several 
short-course  instructors  (DTIC,  UDRI)  expressed  concern  about  inadequacy  of  the 
exposure  for  adequate  learning.  On  the  other  hand,  periods  of  several  hours 
appear  sufficient  for  H.I.S.  trainees  and  possibly  for  CRS  trainees.  Complexity 
of  the  databases  and  the  command  language,  and  perhaps  most  importantly  the 
learning  purpose  of  the  trainee,  can  materially  affect  the  amount  of  exposure 
perceived  to  be  sufficient. 

At  present,  only  LIRS's  ONTAP  programs  are  substantially  location 
independent  (dependent  only  on  the  availability  of  a  terminal).  Of  the  instruc¬ 
tor-based  courses,  LIRS's  evidence  the  greatest  mobility.  In  addition  to 
offering  courses  on  a  regular  schedule  at  field  offices  and  other  facilities 
t'  r oughout  the  U.S.,  LIRS  provides  training  at  client-designated  sites  for  a 
fee.  (Many  public-access  database  producers  also  offer  short  courses  periodi¬ 
cally  in  major  cities  in  the  U.S.  but  less  extensively  than  LIRS).  Although 
NLM  had  provided  abbreviated  introductory  and  advanced  courses  at  Regional 
Medical  Libraries  (RML)  at  least  quarterly,  travel  and  personnel  constraints 
may  limit  this  in  the  future.  NLM  exerts  a  high  degree  of  quality  control 
over  its  courses.  Through  1980,  it  had  authorized  only  one  institution,  the 
RML  at  the  University  of  California,  Los  Angeles,  to  provide  training  recognized 
as  equivalent  to  that  given  by  NLM  staff.  (NLM  and  UCLA  trainees  receive 
Continuing  Education  credits  from  the  Medical  Library  Association.)  One  or 
more  other  institutions  may  receive  NLM  authorization  in  1981.  Some  instruc¬ 
tors  don't  favor  travel  because  of  the  added  burden  of  fatigue  on  an  already 
stressful  and  fatiguing  task.  Other  instructors  regret  their  inability  to 
bring  training  out  of  the  artificiality  of  a  classroom  to  the  terminals  and 
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actual  work  environments  of  individuals  where  they  think  learning  and  retention 
might  be  greater. 

Undoubtedly  some  people  who  have  wanted  to  attend  courses  that  are  given 

at  only  one  or  a  few  locations  have  been  unable  to  do  so  because  of  travel 

expenses.  Even  when  there  is  no  course  fee,  as  in  six  of  the  instances  in 

this  study,  travel  costs  can  range,  conservatively,  from  $100  to  $500  for 

transportation,  $20  to  $45  per  day  for  hotel/motel  accommodations,  and  $20  to 

$30  per  day  for  food.  The  LIRS  arrangement  that  brings  one  instructor  to  any 

location  for  $300  for  a  half  day,  $375  for  2  half-days,  and  $500  for  1^  days 

is  more  cost  effective  if  (sometimes  a  big  "if”)  the  training  site  is  adequately 

equipped.  The  ONTAP  mode  of  learning  may  be  the  most  cost  effective  for  many 

students  when  it  has  been  extended  to  a  larger  number  of  databases.  In  a  test 

of  ONTAP  CHEMNAME,  university  chemistry  students  needed  20  hours  to  complete  a 

Chemical  Abstracts  Service  ( CAS) -produced  course.  Expenses  included  $20  for 

the  CAS-prepared  manual,  $60  for  4  hours  of  connect  time  (at  $  15/connect-hour) , 

and  the  costs  of  CAS's  printed  and  microform  search  aids}  the  latter  aids  are 

4 

reference  materials  that  can  be  shared  by  many  students.  LIRS  gives  2  hours 
of  free  access  to  ONTAP  files  to  those  who  attend  the  $65  introductory  course. 

2a.  PROGRAM  PREPARATION  COSTS 

Program  preparation  refers  to  all  of  the  activities  that  are  involved 
in  determining  the  scope  of  an  education  program,  the  specific  content  of  each 
segment,  the  sequence  of  the  various  segments,  and  the  instructional  materials 
and  aids  to  be  employed  in  each  of  the  segments.  To  teach  the  use  of  online 
databases,  a  considerable  amount  of  time  is  usually  spent  in  developing 
illustrative  search  strategies  and  portions  of  search  strategies  to  demonstrate 
particular  features  of  a  command  language,  one  or  more  databases,  or  both  ol 
these  elements.  The  selection  of  appropriate  strategies  for  the  first  time 
(for  a  new  manual)  can  be  particularly  time  consuming  because  file  responses 
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may  differ  from  expectations,  necessitating  new  search  strategy  development. 
Instructors  usually  encounter  some  system  down  time  during  manual  preparation 
that  prolongs  the  material  preparation  time. 

The  initial  and  update  preparation  times  and  costs  presented  in  Table  I 
are  believed  to  be  very  conservative.  The  code  "est."  is  used  for  data  that 
were  not  directly  provided  but  were  inferred  from  other  information  that  was 
supplied  or  from  the  investigator's  estimate  of  times  or  salary  levels.  It 
is  believed  that  the  nature  of  program  preparation  tasks  is  such  that,  unless 
they  are  timed  as  they  are  performed,  recall  about  times  is  far  from  accurate. 

An  effort  has  been  made  to  limit  the  tabulated  times  and  costs  to  professional 
and  technical  work,  specifically  excluding  typing  and  printing  times  and  costs. 
The  CRS  contract  costs  were  not  subdivided  and  undoubtedly  include  overheads 
and  clerical  costs  as  well  as  professional  costs.  One  might  conservatively 
double  costs  of  the  other  programs  before  comparing  them  with  CRS  costs 
(doubling  CRSin-house  costs  as  well).  For  Pitt  and  ONTAP,  times  and  costs  are 
given  only  for  the  production  of  the  syllabus  and  printed  instructional 
materials;  the  costs  of  the  programming  to  produce  Pitt's  DIALOG  and  ORBI" 
simulators  and  LIRS's  ONTAP  software  are  not  known.  At  Pitt,  one  gifted  under¬ 
graduate  computer  science  student  was  credited  with  writing  and  maintaining 
the  simulators. 

The  data  suggest  that  program  planning  and  development  plus  the  produc¬ 
tion  of  reasonably  comprehensive  written  manuals  take  about  1  to  1  2  months  for 
each  content  focus,  for  example,  instruction  primarily  about  a  command  language 
(that  introduces  database  content  mainly  to  illustrate  features  of  the  language) 
or  instruction  about  one  database  (that  introduces  elements  of  a  command 
language  to  illustrate  retrieval  options  from  the  database).  When  work  is 
coordinated  among  several  people,  an  additional^  to  1  month  may  be  needed. 

When  the  instruction  encompasses  a  command  language  and  one  or  more  databases. 
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the  preparation  time  seems  additive.  For  example,  DTIC  used  8  man-months  lor 
a  command  language  and  3  databases  and  Pitt  used  6  man-months  for  DIALOG, 

ORBIT,  and  a  general  coverage  of  LIRS's  and  SDC' s  databases.  When  the  data¬ 
bases  are  complex,  the  preparation  time  can  be  longer,  e. g. ,  3  months  for 
UDRI's  Profiles  course.  When  the  command  language  and  the  databases  are  simple, 
some  time  compression  can  be  achieved,  e.g. ,  \\  to  2  months  for  language  plus 
database  in  H. I. S.'s  system. 

Not  unexpectedly,  CA1/CBI  requires  more  initial  development  time. 

ONTAP  is  only  minimally  a  CAI  program.  By  providing  the  student  with 

-  Number  of  citations  in  the  answer  set, 

-  Number  of  citations  found,  and 

-  Recall  and  precision  percentages  for  search  strategies 

the  student  creates  for  "canned"  questions, 

the  program  helps  the  student  learn  how  to  achieve  high  recalls  and  precisions. 
Underlying  the  program  is  work  by  LIRS  staff  and  15  professional  librarians 
to  develop  and  refine  29  test  questions  and  to  arrive  at  a  consensus  about 
relevant  citations  retrievable  from  a  test  file  of  32,120  ERIC  records. 

CRS  has  used  the  services  of  a  commercial  firm,  Control  Data  Corp. 

(CDC),  to  obtain  a  true  CBI  program.  CDC  markets  PLATO  (Programmed  Logic 
and  Automatic  Teaching  Operation),  a  system  developed  in  the  mid- 1960's  at 
the  University  of  Illinois.  Several  other  CBI  systems  are  also  on  the 
market;  for  example,  H.I.S.  is  currently  experimenting  with  Boeing  Computer 
Services'  SCHOLAR/TEACH  3.  The  CRS  program  now  in  use  is  a  rewrite.  The 
first  version  was  wordy,  long,  and  required  little  student  interaction. 

With  redirection  supplied  by  CRS  staff,  the  revision  is  subdivided  into 
modules,  each  of  which  can  be  completed  in  8  to  15  minutes,  and  requires 
considerable  intellectual  participation  (rather  than  just  button  pushing). 

The  $100,000  cost,  however,  is  ten  times  that  of  most  of  the  other  instruction 
programs  and  twice  that  estimated  for  the  ONTAP  ERIC  program  (ONTAP  CHEMNAME 


63 


may  be  more  costly  than  ONTAP  ERIC). 

3.  RESOURCES 

Student/Instructor  ratio  is  one  of  the  few  criterion  variables  that 
is  inherently  quantitative.  Does  it  have  predictive  value  for  course  effect¬ 
iveness?  The  best  answer  is  that  this  is  not  now  known.  In  the  DTIC,  NLM, 
and  Pitt  situations,  and  partially  for  UDRI,  the  upper  number  of  students 
is  determined  by  available  terminals  for  hands-on  practice.  Pitt  enlarged  its 
terminal  capacity  in  early  1981  and  is  planning  on  experimenting  with  a 
larger  class  size. 

Only  the  Pitt,  ONTAP,  and  CRS  programs  provide  each  student  with  a 
terminal.  H.I.S.  could  readily  redesign  its  format  to  a  one  person-one 
terminal  mode  but  finds  that  trainees  are  more  comfortable  working  in  pairs. 
DTIC,  NLM,  and  UDRI  are  less  flexible  in  format  redesign,  but  instructors 
indicate  participant  satisfaction  with  the  2  or  3  per  terminal  arrangement. 
Observation  at  Pitt  and  personal  experience  at  GSLIM  suggest  that  many  people 
prefer  the  one-to-one  arrangement  if  given  the  option  after  they  have  gained 
experience  with  the  system  and  databases.  Equipment  and  online  costs  that 
this  arrangement  could  generate  may  prevent  its  use  except  when  terminals 
are  online  to  internal  computer  systems. 

The  TI  Silent  700  continues  to  be  a  popular  terminal.  Notably,  NLM 
and  Pitt  terminals  are  not  equipped  with  CRTs  although  the  instructor's 
terminal  at  Pitt  has  a  CRT  and  the  training  room  has  3  TV  monitors  for  online 
demonstrations.  H.I.S.  and  CRS  terminals  have  no  printers.  Most  H.I.S. 
clients  have  single  item  questions  for  which  they  need  to  knew  answers,  but 
they  have  no  need  for  a  written  record.  Printers  for  them  were  said  to  be 
distracting.  They  are  informed  of  a  print  capability  to  which  their  terminals 
are  linked  should  a  printout  be  desired.  Users  of  CRS's  PLATO  program  receive 
an  instruction  booklet  that  lists  the  retrieval  capabilities  of  the  databases. 
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this  may  be  adequate  or  more  printed  materials  may  be  found  necessary  with 
usage  of  the  program.  I  found  myself  recording  the  content  of  screens  in  a 
notebook  and  recognizing  the  importance,  after  the  terminal  experience,  of 
information  I  didn't  record.  H.I.S.  and  a  segment  of  CRS's  trainees  differ 
from  those  of  other  programs.  Except  for  some  users  of  the  CHEMNAME  ONTAP 
program,  most  students  of  the  other  programs  are,  or  expect  to  be,  working 
as  intermediaries  for  end-users.  By  contrast,  most  H.I.S.  and  CRS  students 
are  end-users.  These  end-users  want  to  learn  only  enough  to  be  able  to 
retrieve  needed  information. 

Only  GSLIM,  Pitt,  and  NLM  give  evidence  of  having  invested  time  and 
money  in  classroom  design,  even  tho  these  facilities  share  with  the  others 
a  starkness  and  overcrowding.  LIRS  and  DTIC  give  courses  in  the  field  and 
are  subject  to  available  facilities  that  can  have  fewer  terminals  and  closer 
quarters  than  their  home  classrooms.  It  appears  to  be  a  tribute  to  the  topic 
being  taught  that  so  many  people,  instructors  as  well  as  students,  have  been 
willing  to  tolerate  poor  facilities  with  little  complaint. 

Specially  prepared  manuals  or  less  ambitious  guides  or  booklets  are 
produced  in  all  of  the  programs.  The  sentiment  was  repeatedly  expressed 
that  printed  materials  that  students  could  refer  to  on  the  job  are  an  essential 
regardless  of  other  media  that  may  be  employed.  DTIC,  NLM,  UDRI,  and  Pitt 
have  produced  the  most  extensive  manuals/workbooks.  NLM's  two  Workbooks  for 
its  introductory  and  advanced  courses  are  by  far  the  most  comprehensive. 

NLM's  objective  was  to  reduce  students'  need  to  take  in-class  notes  to  a 
minimum  so  that  they  could  concentrate  on  the  lecture  content.  DTIC  and  NLM 
also  distribute  other  publications  (e.g.,  thesauri,  searcl  aids)  that  are 
needed  to  learn  their  respective  databases.  LIRS,  GSLIM,  and  Pitt  include 
references  to  database  thesauri,  vocabularies,  and  search  aids  in  their 
instruction;  usage  of  these  materials  is  part  of  GSLIM' s  and  Pitt's  instruc- 


-  65 


tion,  but  students  must  independently  purchase  these  auxiliary  materials. 

By  contrast,  the  materials  H.I.S.  and  CRS  now  provide  are  much  shorter  than 
the  manuals  that  were  initially  distributed;  they  contain  only  database 
content  and  searching  essentials.  The  content  modification  is  based  on 
client  acceptance-testing. 

Most  of  the  manuals/workbooks  are  instructor  transf errable  in  the 
sense  that  instructors  other  than  those  who  prepared  the  manuals  could  readily 
use  them  and  provide  instruction  that  was  consistent  in  coverage  and  content. 
The  brevity  of  LIRS  materials  requires  a  lesson  plan  auxiliary  to  the 
materials.  To  use  UDRI  materials,  the  instructor  must  have  independent  know¬ 
ledge  of  the  FTD  databases  that  have  structural  differences  from  bibliographic 
files  and  are  complex.  Individuals  not  privy  to  rationales  behind  CRS's 
PLATO  programs  may  not  be  able  to  provide  satisfactory  assistance  all  the 
time  to  students  who  press  the  human  HELP  button. 

NLM  and  Pitt  minimize  material  preparation  costs  and  maximize  currency 
of  their  materials  through  the  use  of  an  information  technology.  NLM* s  work¬ 
books  are  produced  with  Vydec  word  processors.  Pitt  uses  the  VAX  computer, 

NLM  instructors  review  the  text  approximately  every  two  months  for  currency. 
The  Pitt  review  is  approximately  monthly.  Both  organizations  print  only  the 
number  of  copies  they  expect  to  need  for  courses  during  the  period.  Thus, 
they  have  few  outdated  copies  that  need  to  be  discarded  or  updated.  Addi¬ 
tionally,  both  use  the  looseleaf  format  so  that  last-minute  insertions  can 
easily  be  made  and  to  enable  students  to  amend  their  texts  or  insert  auxiliary 
materials. 

4.  EVALUATION /FEEDBACK  MECHANISMS 

Minimally,  two  parties  in  an  education  situation  want  a  measure  of 
course  effectiveness.  The  student  wants  to  know  whether  he/she  has  learned 
what  should  have  been  learned;  this  relates  to  course  objectives  and  is  one 


important  reason  [or  performance-measurable  objectives.  The  instructor  wants 
to  know  whether  the  content,  the  method  of  presentation,  time  allocations  to 
various  topics,  etc.,  achieved  the  intended  information  transfer.  Frequently, 
host  institutions  and  sponsors  want  students'  perceptions  of  the  adequacy  of 
the  instructor( s),  course  content,  and/or  the  study  facility/equipment/ 
computer  system,  etc.  All  parties  are  interested  in  effectiveness  --  how 
well  the  program  achieved  its  goals. 

The  most  prevalent  instrument  that  has  been  used  for  evaluation  in 
education  is  the  written  test  administered  to  the  student.  Except  for  the 
university  course,  no  such  test  that  is  submitted  to  the  instructor  for  grading 
is  used  in  online  instruction.  All  but  the  LIRS  courses  give  students  various 
degrees  of  opportunity  to  assess  their  own  progress  through  the  assignment  of 
problem  queries  for  which  students  individually  or  in  groups  develop  search 
strategies  that  they  test  in  online  practice.  Most  instructors  view  this 
mode  of  instruction  as  essential  to  retentive  learning.  It  has  no  substitute 
for  building  student  confidence  and  competence.  NLM' s  MEDLEARN  most  resembles 
an  academic  test  but  it  is  self-administered.  It  is  a  CBI  program  that 
students  accepted  for  NLM* s  introductory  course  are  expected  to  complete  before 
arriving  at  NLM  or  UCLA.  MEDLEARN  is  primarily  expected  to  give  students 
experience  in  logon/logoff  procedures  and  some  familiarity  with  online  data¬ 
bases.  Students  receive  special  passwords  and  free  time  for  MEDLEARN  and 
can  obtain  telephone  assistance  if  needed. 

Six  of  the  programs  distribute  questionnaires  to  students  at  the  end 
of  the  course  that  request  evaluative  information.  The  instruments  for  DTIC, 
LIRS,  NLM,  UDRI,  and  Pitt  are  presented  in  Appendix  B.  For  additional  compari¬ 
son,  forms  are  also  included  that  are  used  by  the  American  Society  for  Infor¬ 
mation  Science  (ASIS)  for  short  courses  and  preconference  tutorials  and  by 
the  U.S.  Government  for  training  course  programs  (Optional  Form  170).  DTIC's 
questionnaire  and  those  of  ASIS  and  the  Government  are  the  most  formal. 
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Several  of  UDRI's  questions  are  the  most  specific.  Questions  3  and  4 
of  the  "Basic  Class  Evaluation"  and  the  first  page  of  "Online  Profile  Evalua¬ 
tion"  questions,  for  example,  probe  details  of  the  command  language  and 
search  features.  DT1C' s  questionnaire  is  at  about  the  degree  of  specificity 
of  the  remaining  questionnaires.  The  5-level  rating  scale  used  in  the  ASIS 
questionnaire  is  often  considered  preferable  because  people  usually  perceive 
a  finer  gradation  than  the  three  options  DTIC's  questionnaire  permits.  The 
ASIS  questionnaire  seeks  evaluative  information  at  a  slightly  more  detailed 
level  than  DTIC's,  and  in  some  instances  the  wording  is  more  "user  friendly." 

To  the  teacher,  an  unsatisfying  aspect  of  teaching  oftei.  is  the 
absence  of  a  longer  term  feedback  mechanism.  L1RS,  DTIC,  NLM,  and  occasionally 
H.I.S.  instructors  get  such  information,  though  not  directly  and  irregularly, 
from  persons  at  the  customer  service  desks  of  these  organizations.  The  NLM 
arrangement  probably  provides  the  most  feedback  because  members  oi  the  MEDLARS 
Management  Section  perform  all  Section  activities  including  manning  the  service 
desk  and  teaching.  This  distribution  of  tasks  may  change  in  1981,  teaching 
being  assigned  to  2  or  3  persons,  but  the  physical  proximity  of  Section  members 
is  likely  to  retain  the  feedback  flow. 

User  group  meetings  provide  another  mechanism  for  feedback.  User  groups 
have  developed  among  DTIC  and  NLM  clienteles.  Each  of  the  groups  has  regional 
quarterly  meetings  and  an  annual  meeting  at  the  central  location  \OilC  and 
NLM,  respectively).  The  Update  programs  sponsored  by  LIRS  several  times 
annually,  usually  betore  or  after  the  annual  meeting  of  a  major  professional 
society  concerned  with  online  databases,  are  similar  to  user  group  meetings. 

The  meetings  are  effective  two-way  communication  mechanisms.  The  database 
producer  or  vendor  disseminates  new  system  information  and  "helpful  hints" 
type  material  anil  database  users  share  experiences  and  discuss  operating  prob¬ 
lems.  Suggestions  presented  at  these  meetings  have  effected  search  system 
changes  as  well  as  changes  in  instruction  programs. 
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(1)  Purchase/Rental 
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Usefulness  Probably  positive  I  N/A  I  Positive  Positive  H/A  N/A 


CRITIQUE  OF  THE  PROGRAMS 


One  program  can't  be  considered  better  or  worse  than  another  because  of 
the  differences  among  them.  Implicitly  or  explicitly,  they  have  different 
training  purposes.  They  address  different  command  languages  and  different 
databases.  Often  they  are  directed  at  different  target  groups.  Experience 
from  the  programs,  however,  does  yield  guidelines  that  can  be  generally  helpful 
for  future  program  development. 

It  appears  that  two  essential  elements  of  any  instruction  program  are 
printed  materials  and  hands-on  practice.  User  need  for  manual  specificity, 
depth,  and  written  analysis  can  vary  widely.  H.I.S.  and  CRS  experience 
should  be  a  reminder  that  brief  is  beautiful  for  some  user  groups.  A  serious 
problem  arises  in  a  situation  like  UDRI's  where  database  complexity  dictates 
a  comprehensive  manual  but  students  lack  experience  to  cope  with  such  a  text. 

The  question  "What  is  enough  hands-on  time?"  is  not  an  issue  in  the  ONTAP  and 
CRS  programs  (if  money  isn't  a  problem  for  ONTAP  students)  because  the 
programs  are  self-paced.  Pitt  believes  that  the  10  to  12  hours  of  practice 
its  students  receive,  accompanied  by  instructor  assistance  as  needed,  enables 
students  to  perform  independently  in  their  own  work  environments.  GSLIM, 

DTIC,  and  NLM  experiences  indicate  that  2  hours  per  student  or  2- student 
group  is  not  adequate  if  database  learning  is  involved;  4  to  5  hours  may  be  on 
the  borderline  of  adequacy. 

It  is  fashionable  to  frown  on  lectures  as  an  antiquated  mode  of  pre¬ 
sentation.  Its  pervasiveness  is  probably  partially  attributable  to  instructors' 
lack  of  familiarity  with  other  media,  but  also  partially  because  of  the 
additional  investment  of  time  (and  hence  cost)  required  to  incorporate 
slides  or  videotapes,  tor  example,  into  instruction  programs.  The  objective 
to  be  accomplished  and  the  audience  must  be  kept  in  mind.  CRS's  14-minute 
introductory  videotape  doesn't  have  the  sophistication  of  a  CBS  documentary 
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but  it  is  informative,  holds  attention,  and  accomplishes  the  objective  of 
introducing  the  viewer  to  SCORPIO  databases.  Videotapes  could  communicate  the 
search  process  and  system  responses  much  better  than  lectures.  The  use  of 
instructional  television  (ITV)  is  widespread  and  is  likely  to  be  as  effective 
in  teaching  the  use  of  online  databases  as  it  is  in  other  areas.  For  content 
subject  to  change  and  when  portability  (e.g. ,  borrowing  by  students,  exchange 
with  other  institutions)  is  desired,  tape/slide  modules  could  be  developed. 

A  comprehensive  British  evaluation  ot  tape/slide  guides  for  library  instruc¬ 
tion  recommends  this  methodology  as  low  in  cost,  easy  to  use  in  program 
preparation,  easy  to  operate,  and  robust  in  handling.  Sound  tapes  alone 
are  being  tested  by  H.I.S.  for  self-paced  learning;  listening  is  integrated 
with  a  printed  manual  and  hands-on  practice.  Slides  are  being  effectively 
used  by  UDRI  to  reinforce  or  illustrate  lecture  content.  The  slides  range 
from  prose  to  tables  to  line  drawings  and  make  good  use  of  color  and  different 
type  sizes.  In  all  instances  where  a  technology  is  used,  it  is  not  used  in 
isolation.  As  is  suggested  in  the  British  study,  "it  may  be  that  interaction 
between  tutor  and  student,  but  in  a  structured  setting,  is  still  the  most 
important  ingredient."  The  lecture  isn't  obsolete,  but  an  online  instruction 
program  that  relies  on  lecture  alone  for  more  than  30-35%  of  the  contact  time 
with  students  probably  needs  to  be  rethought. 

Although  the  computer  has  had  the  potential  to  individualize  instruction 
since  the  mid-195C's,  and  although  CAI/CBI/CMI  systems  have  been  developed, 
utilization  is  still  limited. ^  A  tabular  summary  of  the  major  systems  currently 
available  to  course  writers  is  given  in  Table  II  (page  78).  In  an  excellent 
review  of  the  CAI  state-of-the-art,  Jurg  Nievergelt  cautions  that  more  skill 
is  required  to  use  the  medium  effectively  than  is  required  by  any  other 
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medium.  He  estimates  that  an  author  must  plan  to  spend  between  10  and  100 
hours  to  produce  one  hour  of  student  contact  time,  the  realistic  average 
for  several  authors  working  collaboratively  being  100  hours.  He  anticipates 
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TABLE  II.  MAJOR  CAI/CBI  AND  CMI  SYSTEMS  IN  CURRENT  USE" 


[ - 

tYPE  Of 


PROJECT 

DEVELOPER 

TYPE 

DESCRIPTION 

COMPONENTS 

SERVICES 

STUDENT 

CURRICULUM 

In- 

itflKUOAi 

m-hmiwi  System 

(AIMS) 

New  York  Institute 
of  Technology 

CMI 

Uses  any  course 
materials  thilgned 
for  behavior* 
objectives 

objectives,  stu¬ 
dents,  uu  mutism, 
and  faciUtios. 
instruct  mnaf 
strategies.  tom, 
and  feedb—R 

Evaluation  of 
student  program, 
prater  muons, 
and  empirical  vali¬ 
dation  and  optmti- 
tation  of  instruction 

Any 

Math 

Whconun  System  of 
Instructional  Miiugr 
ment  (WIS-SIM)  designed 
foe  Individually  Guided 
Instruction  (tG£) 

Wisconsin  Research 
and  Development 

Center  for  Cognitive 
Learning 

CM* 

Utilizes  spedfk 
instructional  ob 
tec lives  throu* 
instructional  pro¬ 
gramming  mod*  for 
individual  uudepl, 
establishes  mastery 
levels 

Instructional  pm 
pammmg  mod*, 
measurement  and 
evaluation  mod*, 
and  multi-unit 
school 

O  tier  ion -referenced 
tests,  achievement 
profimg.  diagnosis, 
prescription,  and  in¬ 
struction 

Elementary 

Reading, 

Math, 

Science 

Individually  h»- 
scribed  ?  rructionf 
Management  hdownU— 
System  (IPI/MIS) 

Learning  Research 
and  Develop  men! 

Center  at  University 
of  Pittsburgh  fitted 
by  Research  for 

Seiler  Schools 

CMI 

Uses  IP!  curricular 
materials  which  cm* 

Ploy  detailed  m 
struct  tonal  objec¬ 
tives,  methods,  and 
materials,  estab¬ 
lishes  mastery  level 

Detailed  object¬ 
ives.  methods,  and 
materials,  deter¬ 
mination  of  in- 
dhriduai  compe¬ 
tence,  continuous 
evaluation,  and 
student  perfor¬ 
mance  monitoring 

Diagnoses  and 
prescribes.  Collects 
and  proccsees  infor¬ 
mation,  competence, 
performance,  and  pro¬ 
cess  of  each  student 

Elementary 

Mach 

leuiucwwd  Mwy 

ment  System  (IMS) 

System  Development 

Corporation  South* 
west  Regional  Lab 

CMI 

Adapts  to  any  cm 
Kulum  which  has 
dear,  specific 
learning  objectives 

Frequent  testing, 
profiting,  m* 
vidua!  study  recom¬ 
mendations,  and 
summary  reports 
on  all  tests 

Assnls  with  pacing, 
grouping,  sequencing, 
and  individualization 

Any 

Readtng. 

Math, 

Drama. 

Pubik 

Speaking. 

Art.  Musk 

^rofM  for  Learning 
in  Accordance  writ* 

Nee*  (PLAN*) 

American  Institutes 
for  Research  and 
now  managed  by 

HIM 

nmnifmoi 

Learning  Corporation 

CMI 

Employs  prescribed 
behavioral  objec¬ 
tives  and  Teaching 
Learning  Units 
(TLUs) 

Behavior*  objec¬ 
tives,  Teaching- 
Learning  Units, 
enter  ion -refer¬ 
enced  tests,  gui¬ 
dance,  indi¬ 
vidual  planning 
and  teacher  dne- 

Monitoring  and 
supervising,  test 
scoring,  diagnosis, 
preset ipiions,  in- 
dividualitaiion,  in- 
service  for  teachers 

Any 

m 

Util  ties  ex¬ 
isting  mater¬ 
ials  m  Language 
Arts.  Social 
Studies,  Math, 
Sciences 

Inter  active  Trade- 
•nf  System  (ITS) 

International 

Business  Mtchbtet 
Corporation 

CAI 

Allows  teacher  to 
write  course  with¬ 
out  requiring  de¬ 
tailed  programming 
knoehrfgi 

Course  structuring 
feature,  author 
plans  and  wrttm 
course.  Undent 
sees  ions  may  bo 
scheduled  or  lb 
kon  ai  random, 

to  answer  questions 

Develops  courses, 
teachers  write  cour¬ 
ses,  any  mode  of  pre¬ 
sentations  is  permit- 

led 

Any 

Any  cosme 

Programmed  Lofk  for 
Automatic  Teaching 
Operation  (PLATO) 

University  of 

Illinois,  now 
managed  by  Control 
Oata  Corporation 

CAJ 

In  ferrates  knewledgi, 
fact,  information, 
and  education  devel¬ 
oping  systems  throw#! 
audio  and  visual 
means 

Teacher-designed 
lessons,  evaluates 
and  monitors  stu¬ 
dent  performance 

Any  CAI  mode  can  be 
employed,  develops 
course*  and  units, 
especially  helpful 
with  simulations  and 
game  playing,  revision 
and  editing  at  any  time 

Any 

Utilises  a 
wide  range 
of  subtree 
matter  formats 

Stanford  Project 

Stanford 

University 

CAI 

Use*  prop  ame  and 
materials  with  struc¬ 
tured  curriculum 

Lemons  found  in 
materials,  teacher- 
developed,  monitory 
and  evaluates 
student  performance 

Revisions  and  editing 
at  any  time,  but  for 
problem  solving  and 
drill- praetke,  schools 
usually  contract  for 
servlcaa 

Any 

Applicable  to 
reading,  arith¬ 
metic,  logic . 
algebra,  problem¬ 
solving 

Time-Shared,  Inter¬ 
active,  Coapwer-Coo 
t/oNed,  Information 
TekvHkm  (TKXlT) 

MITRE  Corporation 

CAI 

U  titties  aH  four 
mod*  of  CAI  tit 
provide  besom  for 
students  and  •*- 

HzetmedU 

Design  lemons 
for  individuals, 
employs  ag  lour 

CAI  modes,  eval¬ 
uates  and  moni¬ 
tors  student  per* 

Revision  and  editing 
of  program,  monitor¬ 
ing,  and  evaluating 
student  progress 
utilising  aH  four 
modes  of  CAI 

Any 

Utilizes  a  wide 
range  of  oust¬ 
ing  subject 
matter  formats 

lormiM 


Excerpted  from:  Fred  L.  Sp 1 ittgerber,  "Computer-Based  Instruction:  A  Revolution 
in  the  Making?"  Educational  Technology,  19:1  (Jan.  1979)  20-26. 
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that  the  cost  of  GAI/CBI  programs  will  decline  as  CAI/CBI  becomes  a  mass 
consumption  item  at  which  time  the  cost  may  be  no  greater  than  that  o£  ITV. 

CAI  programs  are  presently  limited  in  the  assistance  they  provide  to  student 
questions.  In  Nievergelt's  words,  "the  limitations  of  what  you  can  program 
show  up  starkly  and  painfully."  CRS '  s  PLATO  program  covers  the  basic  elements 
of  the  SCORPIO  language  and  the  contents  and  retrieval  options  of  four  data¬ 
bases  straightforwardly  and  in  an  interactive  way  with  frequent  questions 
the  student  must  answer  that  holds  student  interest.  Branching  limitations 
are  quickly  evident  in  HELP  responses.  Sophisticated  applications  o£  SCORPIO 
and  complex  search  strategy  possibilities  are  not  included  in  the  programs. 

CRS  has  demonstrated  that  CAI/CBI  can  be  successfully  used  for  an  introductory 
online  instruction  course  but  it  is  not  likely  to  be  realistically  applicable 
to  advanced  content. 

At  probably  half  the  cost  of  CAI/CBI  programs,  the  ONTaP  methodology, 
particularly  as  implemented  by  CAS,  may  be  a  viable  approach  to  giving  students 
who  plan  to  be  or  are  searchers  a  range  of  experience  from  the  introductory 
to  the  advanced  level.  Although  the  16  user  questionnaires  that  were  obtained 
are  too  few  for  definitive  analysis,  more  hands-on  practice  and  more  search 
strategies/case  studies/prototype  searches  appear  clearly  as  desiderata  (see 
Appendix  D).  Po’-haps  other  database  producers  c.an  be  encouraged  to  follow 
the  CAS  lead.  On  he  assumption  that  a  $15/connect-hour  cost  is  quite  reason¬ 
able,  would-be  students  may  have  to  sell  employers  on  the  cost  effectiveness 
of  this  mode  of  learning. 
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I  IMPLICATIONS  FOR  DTIC  INSTRUCTION  PROGRAMS 

This  section  addresses  instruction  program  design  features  that  DTIC 
should  consider  in  formulating  plans  for  future  programs.  The  program  elements 
are  discussed  in  the  order  used  eirlier,  namely 

1.  Purposive  documentation 

2.  Education  program 

3.  Resources 

4.  Evaluation/ feedback  mechanisms 
1.  Purposive  Documentation 

DTIC  should  provide  a  clearer  statement  of  basic  program  goals  for  its 
introductory  and  advanced  courses  than  exist  at  present.  The  existence  of 
these  separate  courses,  augmented  by  several  abbreviated  formats,  indicates 
that  different  end  results  are  expected. 

The  NLM  practice  of  specifying  behavioral  outcomes  expected  of  students 
on  a  lecture-by- lecture  basis  is  initially  laborious  to  do,  but  the  result  is 
beneficial  for  both  instructors  and  students.  Course  designers  and  instructors 
have  a  clear,  unambiguous  direction  that  proves  easier  to  work  with  for  course 
development  than  if  it  were  not  present.  By  phrasing  objectives  in  performance 
measurement  terms  and  sharing  them  with  students,  the  students  know  clearly 
what  they  must  achieve.  The  documentation  also  permits  all  parties  to  the 
instruction  program,  including  external  critics  and  other  organizations  that 
may  wish  to  provide  instruction,  to  review  and  evaluate  it  and,  perhaps,  make 
constructive  suggestions  for  improvement. 

Criteria  --  particular  outcomes  designers  want  to  achieve  --  are 
developed  in  conjunction  with  behavioral  objectives.  Minimally,  they  should 
answer  the  question:  How  much  of  the  performance  attainment  must  be  attained? 
Where  quantities  can  be  specified,  such  as  student/instructor  ratio,  number 
of  hands-on  practice  hours,  number  of  search  strategies  to  be  formulated, 
single  numbers  or  a  numeric  range  should  be  specified.  An  important  facet  of 
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criteria  development  is  that  the  criteria  accurately  characterize  course 
desiderata.  Factors  that  are  measureable  but  are  not  important  to  the  goal 
of  the  course  or  are  not  descriptive  of  student  achievement  have  small  value 
and  should  not  be  employed. 

2.  Education  Program 

The  multi-course  structure  is  a  product  of  both  wisdom  and  necessity. 
Except  for  the  academic  environment,  students  in  other  situations  are  usually 
employed.  They  either  cannot,  or  believe  that  they  cannot,  be  away  from  their 
work  for  more  than  one  week  at  a  time.  Additionally,  short  courses  almost 
always  are  intensive,  requiring  7  or  8  hours  of  continuous  concentration  per 
day.  Pitt's  course  includes  two  2  or  3  hour  evening  sessions  as  well.  People 
become  fatigued  in  these  situations  even  when  the  presentation  is  varied 
through  the  use  of  different  media  and  hands-on  practice.  Strain  among 
students  often  appears  after  to  3  days,  and  the  effect  on  learning  and 
retention  during  a  longer  period,  while  not  known,  may  be  serious.  Thus, 
the  course  subdivision  that  DTIG  now  has  should  be  continued. 

The  content  of  DTIC's  two  regular  courses  and  their  modifications 
bears  re-examination.  The  success  LIRS  has  found  with  modularization  and 
NLM' s  experience  that  will  probably  result  in  optional  modules  at  the  advanced 
level  appears  applicable  to  DTIC,  particularly  since  public  access  is  intro¬ 
ducing  clients  with  different,  less  subject  intensive,  needs  than  those  of  the 
more  homogeneous  DOD  community.  An  introductory  course  consisting  of  a  4-day 
format  is  suggested  to  cover  the  fundamentals  of  the  command  language,  the 
TR  and  CF  report  files,  the  WU  file,  and  the  PP  file.  The  overall  objective 
of  the  course  could  be  to  provide  a  functional  understanding  of,  and  experience 
with,  the  databases  to  enable  students  to  do  query  analysis  and  online  retrieval 
at  their  work  stations. 

DTIC  should  then  consider  the  development  of  advanced  modules  that 
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focus  (a)  on  the  command  language  and  (b)  on  each  of,  or  groups  of,  the 
databases.  The  modules  could  be  1  or  2  days  in  length  as  needed.  A  module 
is  also  needed  for  the  Independent  Research  database.  These  modules  could 
be  offered  back-to-back  over  a  one-week  period  or,  based  on  client  demand, 
some  could  be  offered  more  frequently  than  others.  The  module  approach  is 
suggested  to  more  closely  accommodate  to  client  needs  and  to  indirectly  assure 
the  attentiveness  of  students.  DTIC  might  wish  to  institute  a  requirement 
that  participants  for  advanced  modules  have  3  or  6  months  or  some  other  period 
of  database  and  system  experience  prior  to  enrollment.  DTIC  might  also  wish 
to  prepare  a  self-test  by  which  those  wishing  to  attend  advanced  modules 
could  determine  their  readiness  for  the  modules. 

Cutbacks  in  travel  expenses  can  be  expected  to  more  seriously  impair 
the  ability  of  people  who  should  receive  instruction  from  attending  courses 
conducted  in  one  or  a  few  locations.  On  the  assumption  that  instruction  pro¬ 
grams  will  continue  to  be  lecture  based  for  at  least  the  next  few  years,  the 
number  of  sites  at  which  the  instruction  program  is  made  available  should  be 
appreciably  increased.  Locations  could  be  announced  with  an  understanding 
that  assurances  of  attendance  below  a  certain  number  by  a  certain  date  would 
cancel  the  course. 

A  trend  for  the  past  several  years  has  been  the  development  of  education 
programs  by  database  producers.  FTD  has  elected  a  different  option,  namely 
contracting  for  instruction  program  services.  Since  the  DTIC  databases  are 
not  as  complex  as  several  of  FTD's,  the  problem  of  ensuring  full  contractor 
understanding  of  the  database  structures  and  contents  and  characteristics 
of  the  client  communities  is  less  severe.  The  contract  option  nevertheless 
requires  that  the  contractor  have  a  thorough  knowledge  of  the  DTIC  system 
and  that  a  close  working  relationship  develop  between  contract  and  DTIC 


personnel  so  that  instructors  can  keep  apprised  of  system  changes 


The  issue  of  an  instruction  program  fee  becomes  increasingly  important 
if  instruction  is  to  be  given  extensively  in  the  field.  Insofar  as  DTIC  is 
a  federal  agency  that  wants  the  public  to  utilize  its  databases,  a  high  fee 
seems  inappropriate.  However,  people  tend  to  place  little  value  on  what  they 
get  without  cost.  Therefore  a  modest  fee  of  $50  or  $60  for  the  introductory 
course  and  $30  for  a  2-day  module  seems  reasonable,  particularly  if  locational 
dispersion  reduces  student  travel  costs  and  the  fee  includes  a  workbook  or 
manual. 

3.  Resources 

Instructional  experience  and  student  feedback  strongly  suggest  that 
all  instruction  programs  incorporate  online  hands-on  practice.  The  weight  of 
evidence  points  to  sufficient  terminals  so  that  each  participant  can  have  a 
terminal  or,  as  a  minimum,  a  terminal  can  be  shared  by  no  more  than  2  persons. 
Some  people  are  terminal  shy  and,  even  in  2  or  3  person  groups,  they  avoid 
practice.  This  may  prevent  others  from  seeing  their  slowness  or  mistakes, 
but  this  does  them  a  longer  term  disservice  because  they  may  not  be  able  to 
function  when  they  return  to  their  workplaces.  A  $600  or  $800  or  $1000  invest 
ment  per  terminal  that  may  be  used  by  20  to  30  to  40  people  per  year  in  a 
classroom  reduces  the  per-student  investment  to  nominal  amounts.  Arrangements 
can  be  made  to  rent  terminals  at  field  locations.  Most  students  seem  to  inter 
act  better  with  terminals  having  CRTs,  but  since  many  work-station  terminals 
do  not  have  CRTs,  a  mixture  would  be  more  economical  and  may  be  more  realis¬ 
tically  instructive  to  students. 

The  optimum  student/instructor  ratio  is  not  yet  known.  Experience 
shows  that  20/1  does  not  permit  the  instructor  to  meet  students'  needs.  A 
3/1  or  4/1  ratio  appears  to  make  more  instructor  time  available  than  students 
need  during  most  of  a  course  (though  not  at  the  very  beginning).  A  10/1  or 
12/1  ratio  may  be  the  most  cost  effective  for  DTIC  on  the  assumption  that  most 
trainees  are  already  users  of  an  online  system.  They  are  likely  to  find  the 


DTIC  command  language  a  little  strange  because  it  differs  from  the  one  they 
already  know,  but  they  also  know  what  they  must  learn.  The  DTIC  databases  are 
sufficiently  similar  in  structure  to  other  bibliographic  databases  as  to  again 
permit  most  students  to  learn  through  self-teaching.  In  such  an  environment, 
one  instructor  should  be  able  to  attend  to  all  of  the  questions  posed  by  10  to 
12  students.  This  is  another  element  of  an  instruction  program  where  spreading 
a  resource  too  thinly  may  quantitatively  indicate  cost  effectiveness  but  it  is 
likely  to  impact  the  objective  of  producing  self-reliant  system  users. 

Instructors  uniformly  agree  on  the  value  of  printed  tutorial  materials 
regardless  of  the  instructional  mode  used  for  knowledge  transfer.  Experience 
indicates  that  different  formats  are  acceptable  to  different  audiences.  As 
other  agencies  have  done,  DTIC  has  found  it  advisable  to  produce  both  long  and 
short  versions  of  manuals,  i.e.,  the  "Self-Training  Module  for  Users  of  the 
Defense  RDT&E  On-Line  System"  dated  September  1979,  a  companion  volume  for 
dialup  clients  dated  October  1980,  and  the  "Retrieval  Mini-Manual"  dated 
December  1979  as  well  as  a  quick  reference  booklet,  "Defense  RDT&E  Diverse 
Dial-Up  On-Line  System  Reference  Guide"  dated  October  1980.  All  three  levels 
of  description  and  explanation  are  likely  to  be  needed  in  the  future. 

As  has  been  noted  previously,  DTIC's  future  dialup  clientele  can  be 
expected  to  be  much  more  heterogeneous  than  it  was  when  access  to  DTIC's 
databases  was  limited  to  DOD  agencies  and  contractors  via  dedicated  terminals. 
Some  clients  will  probably  have  little  knowledge  of  technical  information 
systems  and  will  want  to  be  able  to  use  a  few  basic  commands  for  limited 
access  to  the  system.  A  Mini-Manual  type  of  training  and  reference  text  will 
probably  meet  their  needs  as  well  as  a  need  for  quick  verification  of  a  command 
or  a  code  by  more  sophisticated  searchers.  Information  professionals  performing 
as  search  intermediaries  usually  want  as  much  detailed  illustration  of  search 
strategies  and  search  alternatives  as  a  database  producer  or  vendor  is  willing 
to  supply.  This  kind  of  text  could  be  written  so  as  to  be  employed  in 
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conjunction  with  course  delivery  as  NLM  does  with  its  Workbooks.  Alternatively, 
the  detailed  manual  could  contain  material  augmenting  that  covered  during  the 
course  and  possibly  serve  an  encyclopedic  function. 

Use  of  a  variant  of  NLM's  and  Pitt’s  mode  of  producing  printed  materials 
is  highly  recommended.  The  ubiquitous  word  processor  was  initially  designed 
for  this  type  of  production  task. 

The  nature  of  the  course  material  suggests  study  facilities  that  provide 
adequate  table  or  desk  space  for  the  various  manuals,  thesauri,  search  aids, 
and  notebooks  that  must  be  examined,  learned,  and  used  during  the  course  of 
the  instruction  program.  The  amount  of  surface  provided  for  students  in  DTIC's 
current  classroom  requires  that  students  stack  documents  on  top  of  each  other 
and  use  the  floor  and  their  laps  for  overflow.  Furthermore,  chairs  are  less 
than  comfortable  for  long  sitting  periods  and  are  particularly  skimpy  for 
large  men  and  women.  DTIC's  classroom  is  a  good  example  of  an  excessively 
cramped  facility.  The  instructor  has  little  room  to  move  about  for  chalkboard 
or  chart  use.  If  slides  or  transparencies  were  to  be  introduced  into  lectures, 
the  equipment  would  further  reduce  student  work  space  as  well  as  introduce  wires 
likely  to  be  tripped  over.  Functional  designs,  furniture,  and  layouts  are 
being  developed  by  manufacturers  of  office  and  education  equipment.  Whether 
or  not  their  products  are  bought,  they  can  be  examined  for  ideas  that  could  be 
incorporated  into  study  facilities  that  should  be  viewed  as  learning  labora¬ 
tories. 

The  lecture  mode  of  program  delivery  presently  in  use  needs  at  least 
some  integration  with  other  media.  (See  Appendix  C  that  presents  advantages 
and  disadvantages  of  several  instruction  methodologies.)  As  a  minimum, 
command  language  operations,  search  strategies,  and  system  responses  could  be 
exemplified  on  slides  or  transparencies.  Students  should  have  copies  of  these 


visuals  as  separates  or  in  a  workbook  for  note  taking.  DTIC  could  experiment 
with  the  development  of  tape/slide  modules  or  TV  film  for  instruction  about 
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DTIC's  databases.  These  could  be  used  instead  of  lectures  or  could  be  made 
available  for  student  borrowing  as  refreshers  or  in  advance  of  lectures, 
thereby  using  the  face-to-face  time  with  an  instructor  for  question/answer 
discussion.  Computer  systems  often  go  down  during  instruction  when  they  are 
most  needed.  It  is  suggested  that  backup  material  be  developed  on  slides, 
transparencies,  or  film  that  can  adequately  replace  all  online  demonstrations 
should  the  need  arise.  Alternative  student  work  activities  must  also  be 
developed  as  substitutes  for  online  practice  sabotaged  by  down  time. 

CAI  programming  is  not  recommended  for  DTIC  at  this  time  because  (a) 
it  is  too  expensive  to  develop  for  limited  numbers  of  users  and  (b)  it  is 
not  at  the  off-the-shelf  level  yet  for  advanced  users.  The  ONTAP  program 
does  warrant  investigation  by  DTIC.  It  is  suggested  that  DTIC  explore  the 
activities  undertaken  by  the  Chemical  Abstracts  Service  to  implement  ONTAP 
CHEMNAME.  It  is  believed  that  many  information  professionals  who  will  be 
using  DTIC  databases  and  who  need  a  DTIC  refresher,  as  well  as  end-users  in 
DOD,  are  likely  to  find  an  ONTAP-like  program  useful.  Implementation  of  such 
an  instruction  program  could  alleviate  the  need  for,  and  expense  of,  field 
instruction  programs. 

4.  Evaluation/Feedback  Mechanisms 

DTIC  is  already  using  two  important  evaluation  mechanisms,  a  course 
evaluation  form  completed  by  students  and  periodic  user  group  meetings. 

Although  it  might  be  desired  that  the  evaluation  form  more  adequately 
provide  for  written  comments,  it  seems  sufficiently  adequate  to  retain.  The 
main  problem  with  an  evaluation  instrument  administered  immediately  after  a 
course  is  that  students  have  not  yet  had  the  opportunity  to  apply  what  they 
have  learned  in  a  real  situation.  Evaluation  would  be  much  more  useful  if  it 
could  be  obtained  after  the  student  has  used  the  system  for  4  to  6  months. 
Therefore,  it  is  recommended  that  an  evaluation  instrument  be  developed  with 


questions  similar  in  format  to  those  of  ASIS  and  adequate  space  for  written 
comment.  User  group  meetings  may  be  the  best  forums  at  which  to  distribute 
(and  collect)  the  forms.  The  form  could  also  be  a  periodic  insert  in  DTIC's 
newsletter  with  a  short  article  promoting  the  evaluation. 

DTIC  might  also  wish  to  adopt  a  pre-test  mechanism  for  attendees  of 
advanced  courses  similar  to  that  of  NLM.  The  pre-test  is  mailed  to  persons 
wishing  to  attend  one  or  more  of  the  advanced  modules.  It  is  a  means  by  which 
potential  students  can  judge  whether  they  have  acquired  enough  skill  with  the 
system  to  qualify  for  the  advanced  class,  and  it  can  identify  areas  of  weakness. 
The  tests  are  not  sent  to  NLM  for  grading  so  they  do  not  add  to  the  instruc¬ 
tor'  s  workload. 

DTIC  could  systematize  evaluative  information  acquired  at  its  customer 
service  desk  to  feed  back  user  problems  and  experience  to  instructors,  whether 
instructors  are  DTIC  or  contract  personnel.  Ideally,  it  is  believed  that 
instructors  could  benefit  from  spending  even  a  small  amount  of  time  per  month 
in  a  customer  service  capacity.  This  is  the  least  artificial  means  of  obtaining 


feedback  and  evaluative  information  and  it  should  be  utilized 
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INTERVIEW  SCHEDULE 


INFORMATION  SOUGHT  ABOUT  PRO GRAMS /COURSES  THAT  TEACH  ONLINE  USE  OF 
COMPUTER-STORED  DATABASES,  FILES 

1.  What  types  of  courses  are  offered?  (beginning,  intermediary,  advanced) 

2.  What  are  the  durations  of  the  courses? 

3.  When  are  they  given?  (dates,  particular  times  of  the  year) 

4.  What  are  costs  of  the  courses  to  participants? 

5.  Are  courses  subsidized?  By  whom?  By  what  percentage  and  dollar  amount? 

6.  What  are  the  purposes  of  the  courses? 

(what  level  of  competency  is  it  intended  to  provide?) 

7.  What  class  size  is  preferred?  What  is  the  preferable  class  size  limit? 

8.  What  course  materials  are  used?  Who  prepared  them? 

9.  What  teaching  methodologies  are  used? 

(lecture,  discussion,  transparencies,  programmed  texts,  CAI, 
hands-on,  etc. ) 

10.  What  is  the  size,  composition  (background)  of  the  teaching  staff? 

11.  What  are  the  typical  education,  experiential  backgrounds  of  students? 

12.  What  information  content  is  taught? 

13.  How  many  hours  /  What  proportion  of  time  is  allocated  to  hands-on 

learning? 

14.  What  terminals  are  used? 

(keyboard  only,  CRT  +  keyboard,  whose  manufacture,  models) 

15.  How  many  students  are  assigned  per  terminal  during  hands-on  sessions? 

16.  What  kinds  of  assignments  are  performed  during  hands-on  sessions? 

(drill  &  practice,  search  queries,  parts  of  queries,  independent 
project,  etc.) 

17.  Is  a  formal  student  evaluation  performed? 

(copy  of  evaluation  instrument) 

18.  Is  there  a  formal  feedback  mechanism  some  time  after  (6-12  mo.)  the 

program/ course  is  completed? 

(copy  of  feedback  instrument) 

19.  Particularly  if  17,  18  not  done,  how  is  program/ course  effectiveness 

assessed? 

20.  What  assistance  is  given  in  addition  to  courses  (i.e. ,  after  courses) 

to  help  with  problems  students  encounter  in  the  use  of  databases  on 
the  job? 

21.  What  program  changes/improvements  are  being  considered/planned/ 

desired?  Why? 

RWS  9/80 


.ON-LINE  EDUCATION  QUESTIONNAIRE 


I.  Whose  programs  for  teaching  the  online  use  of  databases  have  you  sttended? 


Please  Indicate  vhen  you  had  the  education/training  by  narking 
the  appropriate  boxes.  Please  note  ALL  programs  that  you  have  had. 


Program 

Approximately  how  long  ago  did  you  have  the  program?  | 

In  1980 

In  1979 

In  1978 

In  1977 

In  1976 

Wliir 

1,  BRS  -  Introductory 

2.  BRS  -  Advanced 

3.  Lockheed  -  Introductory 

4.  Lockheed  •  Advanced 

5.  SDC  -  Introductory 

6.  SDC  •  Advanced 

7.  HLM  -  MEDLINE 

B.  LEXIS 

9 ,  OCLC 

10.  DT1C  (formerly  DDC) 

11.  DOE/RECON 

12.  NASA/RECON 

13.  CIST1  -  CAN/OLE 

14.  Cuedre  Associates 

15.  On-the-job,  forme 1 

16.  On-the-job,  informal 

17.  University  -  short 
course  (1  week) 

— 

18.  University  -  short 
course  (2  weeks) 

19.  University  -  regular 
course  (qtr,  semes tr) 

20.  Other  (pis  specify) 

21.  Other  (pis  specify) 

22.  Other  (pie  specify) 

2.  On  a  scale  of  1  to  10,  please  rate  the  programs  you  have  had  on  how  well 
they  prepared  you  to  do  online  searching/retrieval,  by  circling  the 
appropriate  rating  number  below  for  each  program. 

"1"  signifies  very  inadequate  preparation 

"10"  signifies  preparation  that  permitted  you  to  do  searches 

on  your  own  with  a  high  degree  of  confidence  and  proficiency 

Space  is  provided  below  for  ip  to  8  programs.  Please  use  the  numbers 


given  in  the 

left  colusm  of  Question 

1  on 

the 

lines 

below 

• 

Inadequate 

Adequate 

Program  . 

1 

2 

3 

4 

5 

6 

7 

B 

9 

10 

Program  . 

1 

2 

3 

4 

5 

6 

7 

6 

9 

10 

Program  . 

1 

2 

3 

4 

5 

6 

7 

B 

9 

10 

Program  . 

1 

2 

3 

4 

5 

6 

7 

6 

9 

10 

Program  _ 

1 

2 

3 

4 

5 

6 

7 

B 

9 

10 

Program  . 

1 

2 

3 

4 

3 

6 

7 

B 

9 

10 

Program  _ 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Program  . 

1 

2 

3 

4 

5 

6 

7 

6 

9 

10 

3.  What  teaching/presentation  methodologies  do  you  like?  Dislike? 

(e.g.,  lecture,  films,  transparencies,  tape-slide,  role  pitying,  discussion) 
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4.  Whst  aspects  of  Cbe  content*  of  the  programs  did  you  find  particularly  helpful? 


5.  What  program  content  did  you  consider  NOT  adequate?  ( l.e. ,  you  wanted  wore  of) 


6.  What  content  was  NOT  included  that  you  would  have  wanted? 


7.  Bow  many  class  hours  or  workshop  days  do  you  think  are  needed  to  gain  a 
working  knowledge  of 

A  single  database?  ____________  Two  related  databases? _ 


8,  Which  of  the  follcMlng  alternatives  do  you  consider  the  best  to  acquire 
searching  proficiency?  (Please  circle  the  appropriate  letter) 

a.  A  single  course  followed  by  on-the-job  practice 

b.  A  beginning  course,  on-the-job  practice;  6-12  months  later,  an  advanced  course 

C.  A  beginning  course,  on-the-job  practice;  6-12  months  later,  an  advanced 

course  on  the  command  language  and  system  and  advanced  database  courses 
d.  Alternative  a  and  periodic  1-day  updates  on  system  +  databases 

s.  Alternative  b  and  periodic  1-day  updates  on  system  +  databases 

f .  Alternative  c  and  periodic  1-day  updates  on  system  ♦  databases 


9. 


Which  organisations  do  you  think  are  scat  able  to  provide  high-quality  education/ 
training  In  the  online  uae  of  databases? 


Boor 


Database  producers  1  2 
Computer  system  vendors  t  2 
Academic  -  library  schoola  X  2 
Academic  •  adult  education  service  X  2 
Library  netvork/servlce  organisations  1  2 
Consultants,  information  broker*  \  t 
Other  (please  specify)  1  t 


Cood 


-  - 
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10.  What  kinds  of  latching  material s  do  you  prefer? 


11.  Are  you  likely  to  buy  a  self-teaching  annual  Instead  of  attending  a  program? 

.  Yes  .  -  Ho  Maybe 

12.  Are  you  likely  to  pay  for  a  self -paced  cosputer-asslsted  Instruction  program 
Instead  of  attending  a  more  traditional  classroom  program? 

_ Yes  _ Mo  _ Maybe 

13.  Bov  ouch  online  searching  do  you  do? 

Please  circle  the  appropriate  letter  below. 

a.  Over  20  hre/vk  d.  5-10  hre/wk 

b.  15-20  hra/wk  e.  3-5  hrs/vk 

c.  10-15  hrs/vk  f.  %-3  hre/vk 

g.  don't  do  searching 

14.  Bow  frequently  do  you  use  the  vendor  systems  listed  below? 

Please  note  ALL  that  apply 


4  or  more 
hrs/day 

2-4 

hrs/day 

0-2 

hrs/day 

approx  6-8 
hrs/vk 

4-6 

hrs/vk 

2-4 

hrs/vk 

occasion¬ 

ally 

BRS 

Lockheed 

SDC 

OCLC 

MEDLINE 

N  Y  Times 

NASA/RECON 

DOE /RECON 

LEXIS 

WESTLAW 

CAN/OLE 

DTIC 

In-house  files 

- - -  | - 

Other  (*pecify) 

15.  Vhat  type  of  organisation  do  you  work  foil 

a.  For  profit  (not  a  DOD  contractor) 

b.  For  profit  (DOD  contractor) 

c.  For  profit  (lnforoatlon  broker) 

d.  Hot  for  profit  (non-academic) 
a.  Hot  for  profit  (academic) 


f.  Federal  govt,  (non-mllltery) 

g.  Federal  govt,  (military) 

h.  State,  local  government 
1.  Self  esgsloyed 


16.  Do  you  have  any  other  consents,  suggestions  that  could  help  designers  and 
developers  of  future  education/training  programs? 


Please  return  this  Questionnaire  to:  Sevens  Swanson  /  6011  Shaffer  Drive  / 
Alexandria,  Va.  22310  ^  thank  YOU  FOR  YOUR  ASSISTANCE  Hi 
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APPENDIX  B 

COURSE  EVALUATION  QUESTIONNAIRES 


Lockheed  Information  Retrieval  Service  . ....  95 

Defense  Technical  Information  Center  .......  98 

National  Library  of  Medicine  .  100 

University  of  Dayton  Research  Institute  ....  104 

University  of  Pittsburgh  .  112 

American  Society  for  Information  Science  ...  114 

United  States  Government  . .  116 


BBflUDB® 


INFORMATION  RETRIEVAL  SERVICE 
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DIALOG  SYSTEM  SEMINAR 


Name _ _  Phone  

Title/Oi vision _ _  User  No. 

Af-Filiatior.'Cc. _ ___________ 


Wh ere  did  you  learn  about  Lockheed  DIALOG? 

Professional  Meetings  (  )  Advertisements  (  )  Colleagues  {  ) 
Other  (specify)  _ 


Pleas?  indicate  below  your  experience  level  with  online  information  retrieval 
systems. 


DIALOG  0 

to 

10 

hrs. 

10 

to 

25 

hrs . 

over 

25 

hrs 

0 

to 

10 

hrs. 

10 

to 

25 

hrs. 

over 

25 

hrs 

0 

to 

10 

hrs. 

10 

to 

25 

hrs . 

over 

2E 

hrs 

0 

to 

10 

hrs.  __ 

_  10 

to 

25 

hrs . 

_  over 

25 

hrs 

What  subject  areas  are  most  important  to  you? 


B i ored i c i 

n 

Food/Agriculture 

(  ) 

Physical  Sciences/ 

Business 

(  ) 

Government 

(  ) 

Engineering  (  ) 

r  ne*,i  $  try 

(  ) 

Humanities 

(  ) 

Social  Science  (  ) 

Education 

.(  ) 

Patents 

(  ) 

Other 

What  databases  would  you  like  to  see  added  to  the  DIALOG  system? 


(DIALOG,  LOCKHEED 


INFORMATION  RETRIEVAL  SERVICE,  pg.  2) 


Training  Location  _ _ 

Instructor  _ _ Date  _ 

How  would  you  rate  the  content  of  this  session  on  a  scale  of  1  to  1C? 


needs 

improvement 

satisfactory 

good 

excellen 

1  2 

3  4  5  6 

7 

8 

9  10 

How  would  you  ra 

te  the  presentation? 

needs 

improvement 

satisfactory 

good 

exceller 

1  2 

3  4  5  6 

7 

8 

9  10 

What  portion(s) 

of  the  agenda  did  you  find 

most 

useful  ? 

What  suggestions  would  you  make  for  improving  the  session? 


Additiona1  comnents  or,  the  training  session? 


Additional  comments  or  suggestions  for  the  DIALOG  retrieval  service? 


We  appreciate  your  consents,  and  read  them  all.  They  play  an  "important 
in  developing  training  to  suit  your  needs  and  in  continually 
imi'-oving  the  DIALOG  service.  Thank  you  for  your  time. 


(LOCKHEED  INFORMATION  RETRIEVAL  SERVICE) 
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DIALOG  Business  Training  Session 


The  DIALOG  Customer  Services  Staff  is  attempting  to  be  responsive  to  the  training 
needs  of  users.  Your  comments  below  will  assist  us  in  this  endeavor. 

1.  Name:  _ _  DIALOG  Instructor:  _ 

Organization:  _ _  City:  _ 

Phone  No.  _ _  Date:  _ 

User  No. 


2.  Please  indicate  below  your  experience  level  with  online  information  retrieval  systems 


DIALOG 

0  to  10  Hrs. 

10 

to 

25  Hrs. 

over  25  Hrs. 

0  to  10  Hrs. 

10 

to 

25  Hrs. 

over  25  Hrs. 

0  to  10  Hrs. 

10 

to 

25  Hrs. 

over  25  Hrs. 

0  to  10  Hrs. 

_  10 

to 

25  Hrs. 

over  25  Hrs . 

3. 

Please  indicate 

your 

feeling  regarding  the 

level 

of 

today' 

s  session 

by  circling 

a  number. 

1  2 

3 

A  5 

6 

7 

8 

9  10 

Too 

About  the 

_  Too 

Elementary 

Correct  Level 

Advanced 

A. 

Please  indicate 

your 

feeling  regarding  the 

pace 

of 

today's 

session  by  circling  a 

number . 

1  2 

3 

A  5 

6 

7 

8 

9  10 

Delivery 

About  the 

Delivery 

Too  Slow 

Right  Pace 

Too  Fast 

5. 

How  would  you  rate  the  presentation? 

1  2 

3 

A  5 

6 

7 

8 

9  10 

Needs 

Satisfactory 

_  Excellent 

Improvement 

6.  Would  you  be  interested  in  attending  any  DIALOG  Training  Sessions  on  other  topics? 
These  would  be  similar  to  today's  session  in  that  they  would  be  one-half  day, 
would  cover  a  DIALOG  database  or  group  of  databases  and  would  be  taught  by  the 
DIALOG  Customer  Services  staff  (as  opposed  to  a  person  from  the  database  producer) . 
If  so,  please  indicate  your  preference  on  these  topical  sessions  below. 

_  Patents  _  Life  Science  _  Engineering/Physics 

_  Environment  _  Food/Agriculture  _  Education/Social  Science 

_  Other:  _ _ _ 

7.  Any  other  comments  on  today's  session  or  the  DIALOG  service: 


4 


Thank  you  for  your  time.  Please  return  this  form  to  the  DIALOG  instructor. 

Please  continue  comments  on  the  reverse  side. 


(DEFENSE  TECHNICAL  INFORMATION  CENTER) 
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DEFENSE  DOCUMENTATION  CENTER 
DEFENSE  RDT&E  ON  LINE  SYSTEM  TRAINING  EVALUATION 

You  have  just  completed  a  training  program  to  learn  to  operate  the  Defense  RDT&E  On-Line  system.  DDC  is  interested 
in  your  constructive  comments  concerning  course  content,  format  and  presentation  so  that  future  instruction  can  meet  your 
needs.  We  would  appreciate  your  frankness  and  cooperation  in  completing  this  evaluation. 


A.  TRAINING  RECEIVED 

1.  TYPE  OF  TRAINING 

□  RETRIEVAL 
3.  DAT  A  BASE(S) 

□  "R  FILE  (1473) 

□  WU  FILE  (1498) 


PRELIMINARY 


□  INPUT 

O  PP  FILE  (1634) 

□  IR&D  (271) 


2.  LEVEL  OF  TRAINING 

□  BASIC  CD  REFRESHER  CD  INTERMEDIATE  CD  ADVANCED 


4.  LENGTH  OF  COURSE 
CD  3  DAYS 
CD  5  DAYS 


CD  10  DAYS 
CD  2  WEEK! 


B.  YOUR  CURRENT  POSITION 

1.  ORGANIZATION 


CD  GOVERNMENT  CD  CONTRACTOR 


CD  OTHER 


2.  TITLE  OF  YOUR  POSITION 
CD  LIBRARIAN 
CD  RESEARCH  SPECIALIST 


CD  TECH.  INFO.  SPECIALIST 
CD  SCIENTIST 


C.  REMOTE  TERMINAL  INFORMATION  (Preienl  or  on  order) 

1.  EQUIPMENT  ' 


□  U-100 

□  U-200 

3.  OTHER  ON  LINE  SYSTEMS 


O  COP  PRINTER 
□  800  PRINTER 


CD  CLERK-STENO 
CD  PROGRAMEP 


CD  ADMINISTRATIVE 
CD  OTHER 


□  NASA/RECON 


□  DIALOG 


n  SOC 


2.  LOCATION  OF  EQUIPMENT 

□  LIBRARY  □  STAFF  AREA 

□  COMPUTER  ROOM  □  OTHER 


□  ORBIT 


D  OTHER 


4.  WHICH  ONE (S)  CAN  YOU  OPERATE? 


DID  THIS  KNOWLEDGE  DhELP  □  HINDER  YOUR  LEARNING  THE  DDC  SYSTEM? 


A.  TRAINING  MATERIALS 

1.  REFERENCE  GUIDE 


2.  OPERATOR'S  MANUAL 

3.  REFERENCE  TOOLS 


4,  VISUAL  AIDS 

5.  TRAINING  ROOM 


COMMENTS 


COMMENTS  ON  TRAINING 

I  ADEQUATE 


NEEDS  IMPROVEMENT 


B.  OPERATION  OF  EQUIPMENT 

1.  U-100/U-200 _ 

2.  KEYBOARD _ 

3.  PRINTER _ 

4.  SOUND  PAGE  RECORDER 

5.  TAPE  CASSETTE _ 

COMMENTS 


DIFFICULT 


NEEDS  IMPROVEMENT 


C.  SOFTWARE  FUNCTIONS  AND  COMMANDS 

COMMENTS 


□  EASY 


D  difficult 


D  NEEDS  improvement 


PREVIOUS  EDITIONS  ARE  OBSOLETE 


D.  HANDS-ON  PRACTICE 

□  HELPFUL 

Cl  NOT  NEEDED 

COMMENTS 

E.  DATA  BASE  COVERAGE 


l.  tr  file  <1473) 


2.  WUIS  (1498) 


3.  PP  FILE  <1 634) 


4.  IR&D  (271 


ADEQUATE 


NEEOS  IMPROVEMENT 


G.  LENGTH  OF  COURSE 


1.  □  LEAVE  AS  IS  □  LONGER  (NO.  of  days_ 


2,  SUGGESTED  ADDITIONAL  SUBJECTS: 


.  )  □  SHORTER  (No.  of  days 


3.  WHAT  WOULD  YOU  OMIT? 


H.  ADDITIONAL  TRAINING  NEEDED 


}  ‘  ‘  '  ‘  "  —  '  - - 

1.  □  YES 

2.  □  REFRESHER 

3.  □  IN  3  -  6  MONTHS 

□  NO 

O  INTERMEDIATE 

□  IN  6  -  10  MONTHS 

------ 

□  ADVANCEO 

PART  III 

EVALUATION  OF  INSTRUCTOR 

INSTRUCTOR’S  NAME 

DATE 

1.  ORGANIZATION  OF  SUBJECT  MATTER 

□  AOEQUATE 

□  GOOD 

□  NEEDS  IMPROVEMENT 

2.  OELIVERY  OF  SUBJECT  MATTER 

□  ADEQUATE 

□  GOOD 

□  NEEOS IMPROVEMENT 

3.  ATTITUDE  OF  INSTRUCTOR 

□  FRIENDLY 

□  HELPFUL 

□  DISTANT 

4.  ATMOSPHERE 

□  FORMAL 

□  INFORMAL 

5.  PRACTICE  PROBLEMS 

□  HELPFUL 

□  ADEQUATE 

□  NOT  NECESSARY 

COMMENTS 

PART  IV 

EVALUATION  OF  COURSE 

□  AOEQUATE 

□  GOOD 

□  VERY  GOOD 

□  EXCELLENT 
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(NATIONAL  LIBRARY  OF  MEDICINE,  QUESTIONNAIRE  FOR  INITIAL  TRAINING  COURSE) 

ON >L!ME  SERVICES  TRAINING 
Course  evaluation 

The  National  Library  of  Medicine  would  greatly  appreciate  your  consent a , 
suggestions,  and  criticisms  on  Che  following: 

Course  content: 


Sequence  of  topics: 


Uands-or.  exercises  and  follow-up  discussions: 


o 
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(National  Library' of  Medicine,  Initial  On-Line  Services  Training) 


Count  evaluation  •• 


Hand-out* ,  written  extrelaes,  and  other  loatructional  old*: 


Instructors: 


Any  additional  coonenta: 


Name  (optional)  _ ______ 

Cetea  of  training  course 


HI*  1 


(NATIONAL  LIBRARY  OF  MEDICINE) 
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ADVANCED  ONLINE  TRAINING  CLASS  EVALUATION 


We  appreciate  your  comments  on  the  training  you  have  received  at 
NLM  and  any  suggestions  you  may  wish  to  make  for  improvements  or 
changes  in  future  training  courses. 

1.  Course  prerequisites: 


2.  Systems  mechanics-overview: 


3.  Chemical/Toxicological  Searching: 

a.  CHEMLINE 

b.  TOXLINE 

c.  RTECS 

d.  TDB 


4.  Special  application  clinics 
MEDL INE/MeSH 

CANCERLINE: 

SERLINE: 

CATLINE,  AVLINE,  NAF : 


BIOETHICS,  HISTLINE,  HEALTH,  EPILEPSYLINE: 


-  LOJ  - 

(National  Library  of  Medicine,  Advanced  Online  Training  Class  Evaluation) 

Page  2 

6.  Search  formulation  sessions: 


7.  Topics  which  should  be  added  or  omitted: 


E.  Aspects  of  the  course  you  particularly  liked  or  disliked: 


9.  Any  additional  comments  or  suggestions: 


Your  name  (optional)  and  dates  of  training: 
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(University  of  Dayton  Research  Institute)  NAME 

1979  to  Pxd6e.nt  (August  1979  on) 
BASIC  CLASS  EVALUATION 


Please  mark  the  one  response  which  most  accurately  describes 
what  your  major  involvement  with  the  CIRC  II  system  will  be. 
(Check  one) 

rn  ) 

a . 

I  will  search  CIRC  II  to  obtain  my  own  information 
which  I  will  use  for  other  tasks. 

[19] 

b. 

I  will  search  CIRC  II  to  obtain  information  for 
someone  else. 

[  n 

c. 

I  will  use  information  from  CIRC  II  which  someone 
else  obtains  for  me;  I  will  not  search  the  system 
myself. 

[20) 

d. 

I  must  know  something  about  CIRC  II  to  do  my 
job  but  I  do  not  regularly  search  the  system  or • 
use  the  information  it  contains. 

[  5] 

e. 

Other  (please  describe) . 

2.  I  anticipate  my  average  weekly  usage  of  the  system  (for  both 
secure  and  non-secure  sessions)  will  be:  (Check  one) 


I  5] 

a. 

0  hours 

r  J°] 

b. 

less  than  a  half  hour 

[  14] 

c . 

1/2  -  2  hours 

[  9] 

d. 

2-5  hours 

r  n 

e. 

5  hours  or  more. 

1 
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Please  rate  your  level  cf  understanding  cf  the 
following  features: 

(UDRl,  Basic  Class  Evaluation) 


1.  sign-on 

2.  ..off 

3 .  . .off  cent 

4.  .  ..search 

5.  paragraph  searching 
€.  . .select 

T.  cr-er.c-not 
S.  par-sen-aej 
9.  rec*ref ereneinc 

10.  parentheses 

11.  -.site  nixed  operators  with 

bacXreferencing  t 
parentheses 

12.  . .trovse 
12.  newspec 

14.  ..print 

15.  ..nail 

IS.  truncation 
1“.  roct 
IS.  literals 
IS.  ..display 

20.  . .puree 

21.  ..chance 

22.  . .save/. .exec 

23.  ..cement 

24.  . .send/. .reev 

25.  ..help 

26 .  C5A0 
21.  fACl 

28.  cssr 

29.  XLAX 

30.  ?TtN  paragraph 

31.  ?FCN  equates/relates 

32.  ?*tN  relationship  and 

attribute  codes 

33.  using  the  ?FDN  codes 

(searching  with) 

34.  using  the  ?F1N  aicrofiche 


Bad  Scae 
Difficulty  at 

Still  Somewhat  First,  but  Z  Now 
Confusing  Understand 


11 

11 

1L 

11 

26. 

II 

m 

11 

77 


Ji. 

2 

_2 

_2. 

_3 

"~7 

_2_ 

_2_ 

_2_ 

_4_ 

_9_ 

_1_ 

_± 

_£ 

17_ 

11 

11 

21 


J_5 

_22 

11 

11 

1L 

11 

11 

11 

21 

ll 

It 

11 

11 

It 

11 

21 

II 

11 

11 

11 

IS 


22 

11 

15 


IS 

,  IS 
19 


I  Understood 
From  the  Start 


II 

36 

JUL 

II 

11 

11 

21 

23. 

II 

21 


11 

11 

11 

11 

21 

11 

11 

21 

II 
1L 
11 
11 
It 
21 
ZL 
21 

II I 
_6 
_6_ 
_5_ 

-1 

_L 

11 

-I 


2 


n;  .1 


-  106  - 


4.  Ter  these  features  in  Question  #2  which  you  narked  "Still 

Scnewhat  Confusing",  please  list  their  number (s)  in  the  space 
next  to  the  most  accurate  response: 

3  f  7  > ;  28(5);  29(6)  Wasn’t  discussed  in  class. 

22,23,26,27,33(2); 

:  17 lil _ ,  Was  mentioned ,  but  needed  greater  explanation. 

22,23(2};  5,7,8,9,11,  .  ^  , 

3f.  3?  33M1;  31(5) _  I  tried  to  use  it  m  terminal  practice 

problems ,  but  it  didn't  work  well. 


re  tnere 
..eve  been 


any  commands  or  techniques  which  ycu  feel 
covered  in  greater  depth  -in  class? 


should 


3 


(UDRI  Basic  Class  Evaluation) 

-hi ch  aspects  covered  in  the  course  will  be  particularly 
helpful  once  you  are  back  on  the  job?  Were  they  presented 
clearly  anc  completely  enough  that  you  will  feel  at  ease 
using  them? 


a. 


Were  any  of  the  materials  distributed  in  class  particularly 
helpful  or  useless?  Which  ones? 


b 


Should  any  additional  information  be  covered  in  handouts 
that  at  present  is  not? 


(UDR1  Basic  Class  Evaluation) 


'  as 

ate.ness  of  ten.;iral  practice  problems  (check  as 
applicable) . 

y. 

_  a. 

well  placed. 

7 

w 

w  • 

did  not  come  at  appropriate  times  during 

the  course. 

7 

_  c. 

too  simple. 

1 

_  d. 

too  complex. 

26 

_  e. 

level  of  difficulty  was  about  right. 

0 

■F 

• 

too  many  problems. 

S 

_ g- 

net  enough  problems. 

21 

h. 

the  problems  were  fine. 

ZNTS 


vrhat,  ir.  your  opinion,  makes  a  good  searcher?  How  did  the 
training  package  help  you  develop  those  characteristics? 

In  what  wavs  can  it  be  improved? 


(University  of  Dayton  Research  Institute) 
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NAME 


7  979  to  Pfititnt  (November  1979  on) 
ONLINE  PROFILE  EVALUATION 

l.  Please  rate  your  level  of  understanding  of  the  following 


STILL  SOMEWHAT 
CONFUSING 


HAD  DIFFICULTY  AT 
FIRST  BUT  I  NOW 
UNDERSTAND 


rsr'v 

r  •  \\j .  i 


1.  customer 
identification 
information 

2.  initial  profile 
construction 

3.  ..END 

4.  ..SAVE  procedures 


10 


11 


■I  FI  CATION 


o. 


c . 
7. 
S. 
9. 
10. 
11. 
12. 


function  of  PCN 
(profile  control 
number) 

..EDIT 

MODIFY  subcommand 
ADD 

DELETE 
REPLACE 
LIST  . 

INDEX  subcommand 
..OFF  TEMP 


12 


10 


11 


8 

IT 


PROL 


16. 


17. 


use  of  logical 

operators 

use  of  truncation 

document  structure 

a.  PRN  (Profile 
Reference 
Number) 

b.  Comment 

c.  PROFILE 
COPY  function  in 
..EDIT 


11 


10 


11 


features : 


UNDERSTOOD 
FROM  THE  START 


21 


17 

11 

18 


11 

1L 

H 

1± 

11 

11 

11 

11 

20 


22. 

21 

U 


11 

11 

15 


-  no  - 

(UDRI,  Online  Profile  Evaluation) 

For  Those  features  in  Question  1  which  you  marked  "Still  somewhat 
confusing",  please  list  their  number(s)  in  the  space  next  to  the 
most  accurate  response: 


_ Was  not  discussed  in  class 

2(7);  7-1211);  74(7)  Needed  greater  explanation 

7(71;  79(1) _ I  tried  to  use  it  in  terminal  practice,  but  am 

still  confused. 


Are  there  any  commands  or  techniques  which  you  feel  should  have  been 
covered  in  greater  depth  in  class? 


(UDR1,  Online  Profile  Evaluation) 


-  Ill  - 


Were  any  of 
Which  ones, 


i 


the  materials  distributed  in  class  particularly  helpful, 
and  why? 


Were  any  materials  particularly  confusino?  Which  ones,  and  why. 


Please  include  here  any  additional  conments  you  may  have  in  regards  to 
materials  covered. 


3 


(University  of  Pittsburgh  On-Line  Training  Center) 

Evaluation  Form 


We  would  like  to  have  your  personal  opinion  on  various  aspects  of 
the  training  session. 


1.  Presentation  of  material  was: 


Too  fast 


About  right  Too  slow 


2.  Amount  and  frequency  of  review: 


Overdone 


About  right  Inadequate 


3.  Did  you  feel  your  time  on  the  Emulator  was: 


Too  long 


Enough 


Not  enough 


Too  much 
supervision 


Right  amount  Not  enough 

of  supervision  supervision 


4.  Concerning  your  time  on  the  Commercial  system: 


Too  long 


Enough 


Not  enough 


Too  much 
supervision 


Right  amount  Not  enough 

of  supervision  supervision 


5.  Concerning  the  total  duration  of  the  training  period: 


Too  long 


About  right 


Too  short 


-  113  - 


-2- 


6.  I  feel  that  I  can  do  an  adequate  search  for  a  user  on  the 
commercial  system. 


Yes 


No 


Perhaps 


7.  As  a  result  of  this  training  session  I  now  feel  that  with  the  aid 
of  a  printed  manual,  1  could  teach  myself  another  commercial  system: 


Yes  No  Perhaps 

8.  We  would  appreciate  any  comments  or  suggestions  for  the  improvement 
of  the  training  session. 


American  Society  for  Information  Science 
EVALUATION  FORM  FOR  EDUCATION  ACTIVITIES 
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COURSE  TITLE. 


A.  PERSONAL  INFORMATION 

1 )  Current  employment  status: 
Description  of  position _ 


2) 


Years  of  experience  as  an  information  professional _ 

How  would  you  assess  your  knowledge  of  the  material  covered  in  this  course? 
(check  as  many  as  apply) 


Before 

After 

Course 

Course 

□ 

□ 

Unfamiliar  with  the  material. 

□ 

□ 

Familiar  with  the  material  in  a  very  general  sense. 

□ 

□ 

Familiar  with  the  material  and  the  basic  concepts/theories  and/or  methods/techniques 
underlying  it. 

□ 

□ 

Understand  many  of  the  concepts/theories  and/or  methods/techinques  underlying  the 
subject  of  this  course  and  able  to  apply  them  in  practice. 

.substandard 


B.  GENERAL  ARRANGEMENTS 

1)  How  would  you  rate  the  pre-course  information  and  orientation?  (Circle  only  one  number) 

1  2  3  4  5 

- 1 - _j - 1 - J - »- 

Poor  Average  Excellent 

2)  How  would  you  rate  the  facilities  (room,  living  accommodations,  food)  provided  for  the  course? 

_ outstanding  _ comfortable  _ adequate 

_ substandard  _ not  applicable 

3)  How  would  you  rate  the  audiovisuals,  handouts,  and  other  materials? 

_ outstanding  _ adequate 

C.  COURSE  ARRANGEMENTS 

1 )  Was  the  course  what  you  expected  from  the  announcements? 

_ yes  _ too  elementary 

2)  Was  the  level  of  the  course  suitable  for  your  needs? 

_ about  right  _ too  elementary 

3)  What  is  your  opinion  of  the  total  work  load  of  the  course? 

_ about  right  _ too  heavy 

4)  What  is  your  opinion  of  the  length  of  time  allowed  for  the  course? 

_ _ about  right  _ too  long 

5)  Rate  the  balance  of  time  allowed  for  each  topic.  (Circle  only  one  number) 

1  2  3  4  5 


_too  advanced 


too  advanced 


.too  light 


too  short 


Poor 


Average 


6)  Were  the  presentations  scheduled  in  logical  sequence?  (Circle  only  one  number) 
12  3  4 


Excellent 


Poor  Average  Excellent 

7)  Was  the  variety  of  presentations  sufficient  to  maintain  your  interest?  (Circle  only  one  number) 
1  2  3  4  5 


Poor 


Average 


Excellent 


8)  Was  the  course  practical  in  terms  of  your  needs?  (Circle  only  one  number)  _  ^  3  _ 

1  2  3  4  5 

_j _ 1 _ 1 - 1 - 1 — 

Poor  Average  Excellent 

9)  Did  the  course  provide  adequate  theory  on  the  subject?  (Circle  only  one  number) 

1  2  3  4  5 

_ 1 _ 1 - 1 _ 1 _ 1 _ 

Poor  Average  Excellent 

D.  SPEAKERS  (Rate  the  speakers  on  the  following  aspects  of  their  presentation  on  a  scale  of  1  (''poor”)  to  5  (“excellent") 
with  3  being  "good") 

Command  of  Clarity  and 

Speaker  Preparation  Subject  Organization 


E.  ACHIEVEMENT  OF  COURSE  OBJECTIVES  (Refer  to  the  stated  objectives  of  the  course  and  rate  their 
degree  of  achievement  by  circling  the  number  on  the  scale  that  reflects  your  opinion) 


Not  achieved 


F.  SUMMARY  EVALUATION 


Partially  achieved 


Fully  achieved 


1.  How  would  you  rate  your  satisfaction  with  the  course  as  a  whole? 

1  2  3  4  5 

—  1 _ 1 _ 1 _ 1 _ j 

Dissatisfied  Satisfied  Very  satisfied 

2.  General  comments  (Add  below  any  further  comments  you  may  have  on  any  aspect  of  the  course) 


G.  FUTURE  COURSES 

Are  there  any  other  topics  or  subjects  that  you  would  like  to  see  presented  in  the  ASIS  Continuing  Education  Program? 
If  so.  please  state  below. 


loptional) 

PLEASE  RETURN  TO:  ASIS  CONTINUING  EDUCATION  PROGRAM,  1010  Sixteenth  St..  N.W..  Washington, 
DC  20036 


J 


(UNITED  STATES  GOVERNMENT) 


11' 


REQUEST,  AUTHORIZATION,  AGREEMENT 
AND  CERTIFICATION  OF  TRAINING 


L  Agency,  corn  agency  subelemen!  •)  I  1  I 
ond  submitting  office  number  l — — | 

(uunv/t-u-u-wil  I 


1.  Apphcant's  nam  (Last  Fmt  Middle  Ininal) 

MKAte  preferred  title  Mn.  Mr.  Mi  etc ) 


4.  Home  address  (NMmbrr.  Street,  etty.  State,  ZIP  code) 


'  ^  OtgenuetWi  mawflf  address  ( Branch  non /Office/Bureau /Agency ) 


IV  Position  title/tunction 


Section  A— TRAINEE  INFORMATION 


Inter  first  It)  l  Social  Security  Humber 

5  jotters  ol  l—] 

fait  name 


),  Home  telephone 


Area  code  ;  Humber 


I  —  J  Date  ot 


birth  (  Year  and  month  j 


(example — bom 
January  14.  194) 
shown  as  4J/0I 


%.  Position  level  (Mark  (X)  oat  only l 


a.  Hon  supervisory  J  I  c.  Manager 


4.  Ixecutr* 


II.  Humber  of  pnot. 
nongovernment 
t»e*rwAf  bars 


Section  B— TRAINING  COURSE  DATA 


lie.  Hame  and  mailing  address  of  training  vendor  (So.,  ureei  ary.  Stare.  ZIP  code)  ;  111.  Location  of  training  site  (If  same  mark  taxi 


M.  Course  title  end  training  objectives  (Benefits  to  be  deneed  by  the  Government} 


i  17.  Cel ato«/ Course  Ho 


It-  framing  period  (6  digits) 


II.  Ho  of  course  hours  (4  digits) 


•7  |  H  Trammg  codes  (See  instructions) 


AREAS  OF  EVALUATION 

(Place  (<0  i*  dffeofnate  column  to  indicate  your  evaluation  of  Hems  26  through  )  7  Do  not  attempt  to  spin  a  rating) 


M  Stated  ebtectme 


A  Yes 


Excel  lent 


B  —  Partial  ly 


B  =  Sufficient 


M  Ortmtbtiob  el 

eubtact  matter  a  -  Well  organ i Ted  B  —  Adequate  c  -  Poorly  organized 

M  SurtaMify  of 

metructionei  materials  A  ticeffent  B  =  Adequate  (  -  Poor 

1 

■ 

i 

! 

Ml  Lever  o»  drtficwty 


11  Lmtgth  of  course 


Too  advanced 

B  *  Appropriate 

C  -  Too  Ham  antary 

Too  long 

B  r  Appropriate 

C  -  Too  short 

M  AdpHcaMrfy  af  ml 
matter  to  tfce  rob 


M  faoWre* 


A  Significant 


-  EeceHent 


Mighty  recommend 


*  Yes 


B  -  Adequate 


B  —  Good 


B  Recommend 


8  No 


C  -  mmtnificant 


C  -  Hot  recommended 


t  •  Hof  appHcjbtf 
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APPENDIX  C 

ADVANTAGES  AND  DISADVANTAGES  OF 
INSTRUCTION  METHODOLOGIES 

The  pages  that  follow  reproduce  material  from  the  report, 
Managerial  and  Supervisory  Training  Techniques  and  Methods 
U.S.  Civil  Service  Commission,  Bureau  of  Training,  1977, 
on  the  following  methodologies: 

-  Lecture 

-  Slide/Tape  Presentation 

-  Instructional  Film 

-  Instructional  Television 

-  Self-Paced  Instruction 

-  Coaching 

-  Planned  Exercises 

-  Correspondence  Study 
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Lecture  •  Lecturette 

The  lecture,  an  oral  presentation  prepared  and  delivered  by  a  subject-matter  expert,  is  probably 
the  oldest  and  most  basic  form  of  instruction.  It  is  used  to  supply  the  greatest  amount  of  information  in  the 
least  time.  Like  other  types  of  information  presentation,  it  does  not  allow  for  feedback  from  the  learners; 
thus,  it  should  be  combined  with  participative  techniques  whenever  possible. 

How  It  Can  Be  Used:  1.  To  introduce  concepts,  identify  and  analyze  problems,  or  clarify  issues 

related  to  supervisory  and  managerial  functions. 

2.  To  deliver  training  content  which  is  best  presented  all  at  once  in  an  orderly 
manner  and  does  not  require  practice. 

Who  Can  Benefit  from 

It:  1.  Trainees  who  have  similar  needs  and  similar  capabilities  to  learn  the 

material. 

2.  Managers  and  supervisors  who  are. highly  motivated  to  learn  but  short  of 
time. 

1.  Lectures  are  efficient  in  terms  of  time,  facilities  needed,  and  the  number  of 
participants  who  can  be  trained  at  one  time. 

2.  The  lecturer  retains  control  over  the  learning  content. 

3.  This  technique  may  be  more  acceptable  than  others  are  to  participants  who 
are  accustomed  to  a  traditional  teaching-learning  situation. 

4.  Lectures  can  easily  be  varied  to  suit  trainees'  needs  and  can  be  used  with 
almost  any  other  training  technique. 

1.  Participants  usually  have  no  opportunity  to  make  comments,  ask  questions, 
or  otherwise  show  that  they  understand  the  material  presented. 

2.  The  lecturer  must  be  qualified  both  as  a  subject-matter  expert  and  as  a 
speaker. 

3.  Long  lectures  (over  40  minutes)  may  not  be  readily  learned  or  remembered. 
Lecturettes  are  shorter  and  can  be  highly  effective. 

4.  Listeners  might  not  make  the  transfer  from  intellectual  understanding  to 
practical  application. 

1.  Some  books  dealing  with  supervisory  and  managerial  training  contain 
outlines  for  lecturing  on  pertinent  topics. 

2.  Lectures  are  often  more  effective  if  supplemented  by  visual  materials  such 
as  handouts,  slides,  chalkboard  notes,  etc. 

1.  Analyze  audience  needs. 

2.  Determine  objectives. 

3.  Plan  lecture. 

4.  Present  lecture,  listing  main  points  and  then  expounding  on  them. 

5.  Summarize  main  points. 

For  More  Information: 

International  Labour  Office.  An  Introductory  Course  in  Teaching  and  Training  Methods  for  Management 
Development.  Geneva,  1972. 

Pfeiffer,  J.  William,  and  John  E.  Jones.  The  1972,  1973,  1974,  1975,  1976  Annual  Handbook  for  Croup 
Facilitators.  La  Jolla,  CA:  University  Associates  Publishers,  Inc.,  1974  (Also  1972-76). 

Staton,  Thomas  R,  How  to  Instruct  Successfully:  Modem  Teaching  Methods  in  Adult  education.  New 
York:  McGraw-Hill.  1960. 


Advantages: 


Disadvantages: 


Materials: 


Procedures: 
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The  Slide/Tape  Presentation 

With  the  advent  of  inexpensive,  portable  machines  for  synchronizing  still  visuals  and  sound,  the 
most  basic  audiovisual  system  was  created.  Although  relatively  new,  the  slide/tape  presentation  is  now  in 
wide  use  for  both  group  and  self-paced  instruction.  It  is  used  in  conference  rooms,  in  classrooms,  in 
individualized  learning  centers,  and  for  on-the-job  training.  Several  slide  projectors  can  be  used  with  a 
controller  and  tape  player  for  a  multi-screen  presentation. 

How  It  Can  Be  Used:  1.  To  reinforce  information  presented  orally  with  visual  cues. 

2.  To  stimulate  interest  in  the  topic  being  discussed. 

Who  Can  Benefit  from 

It:  All  levels  of  managers  and  supervisors. 

Advantages:  1.  Slides  and  tapes  can  be  presented  together  or  separately;  for  example,  the 

slides  can  be  used  first  with  the  tape  and  later,  as  the  narrative  changes,  with 
a  lecture. 

2.  After  the  initial  equipment  and  taping  expenses  are  met,  the  presentation 
can  be  repeated  many  times  at  little  cost. 

3.  The  slide/tape  production  equipment  is  relatively  inexpensive. 

1.  The  slide/tape  show  by  itself  does  not  allow  for  discussion,  interaction,  or 
practice  unless  it  is  specifically  designed  to  do  so. 

2.  Two  pieces  of  equipment  must  be  used  together,  possibly  causing  problems 
in  coordination. 

3.  Preparation  of  tapes,  slides,  and  scripts  requires  considerable  expertise;  most 
training  offices  have  to  contract  for  the  service  from  an  outside  source. 

1.  A  slide-synchronizing  tape  player  plays  a  tape  with  prerecorded  advance 
pulses  that  control  a  compatible  slide  projector.  With  non-automatic 
equipment  slides  are  advanced  manually. 

2.  Slide/tape  presentations  for  supervisor  and  manager  training  are  available 
from  firms  which  produce  instructional  media. 

1.  When  used  with  groups,  the  presentation  may  be  introduced,  allowed  to  run 
without  interruption,  and  then  discussed.  If  the  presentation  is  a  long  one. 
the  trainer  may  wish  to  stop  at  intervals  for  a  question  and  answer  period. 

2.  The  slide/tape  presentation  may  also  be  used  with  workbooks  in  self-paced 
instruction. 

Related  Approaches:  Slide/tape  presentations  (sometimes  called  Slide/Sound  Shows)  are  often  used  in 

Programmed  Instruction.  Also,  see  references  for  Audiovisual  Aids  and 
Techniques. 

For  More  Information: 

Bergevin,  P.,  D.  Morris,  and  R.  M.  Smith.  Adult  Education  Procedures:  A  Handbook  of  Tested  Patterns  for 
Effective  Participation.  New  York:  The  Seabury  Press,  1963. 

Otto,  Calvin  P.,  and  Rollin  0.  Glaser.  The  Management  of  Training:  A  Handbook  for  Training  and 
Development  Personnel.  Reading,  MA:  Addison-Wesley,  1970. 

Pula,  Fred  John.  Application  and  Operation  of  Audiovisual  Equipment  in  Education.  New  York:  John 
Wiley  8t  Sons,  Inc.,  1968. 

Rigg,  Robinson  P.  Audiovisual  Aids  and  Techniques  in  Managerial  and  Supervisory  Training  London: 
Hamish  Hamilton,  Ltd.,  1969. 


Disadvantages: 


Materials: 


Procedures: 
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The  Instructional  Film 


The  practice  of  presenting  information  through  films  has  been  widely  accepted  by  the  training 
community  for  many  years.  Films  offer  the  advantages  of  virtually  all  other  media  in  an  easy-to-use  format. 
Films  can  combine  the  benefits  of  other  means  of  presentation  (still  visuals,  print,  lecture,  etc.)  to  enhance 
instructional  content. 


How  It  Can  Be  Used: 


Who  Can  Benefit  from 
It: 

Advantages: 


Disadvantages: 


Materials: 

Procedures: 


Related  Approaches: 


1.  To  demonstrate  ideal  ways  of  doing  such  tasks  as  interviewing  and 
counseling. 

2.  To  illustrate  new  concepts  and  techniques. 

3.  To  take  the  place  of  the  instructor  when  he  or  she  cannot  be  present. 

4.  To  teach  human  relations,  an  important  part  of  supervisor  and  manager 
training. 

Films  can  be  used  in  virtually  any  training  situation. 

1.  Films  standardize  the  dissemination  of  information;  variations  among 
instructors  are  eliminated. 

2.  The  person  who  runs  the  film  can  control  time  and  space  in  ways 
unparalleled  with  other  media— animation,  magnification,  slow  and  fast 
motion,  etc. 

3.  Of  all  information  presentation  techniques,  films  are  the  most  likely  to 
catch  and  hold  the  full  interest  of  the  participants. 

4.  A  lecture,  demonstration,  skit,  or  discussion  that  is  filmed  can  be  used  many 
times. 

1.  Since  the  training  room  usually  must  be  darkened  during  the  film,  the 
instructor  misses  the  trainees'  immediate  reactions  to  it.  Also,  trainees  may 
not  be  able  to  take  notes. 

2.  Equipment  and  facilities  for  presenting  films  are  not  always  available. 

3.  Unless  trainers  are  able  to  contract  for  the  production  of  special  films,  they 
must  rely  on  standard  training  films  in  which  some  of  the  content  may  be 
inappropriate.  Sometimes,  but  not  always,  the  trainer  may  be  able  to  use 
just  part  of  a  film. 

Films  are  available  from  a  number  of  instructional  media  firms.  Most 

Government  agencies  with  training  functions  have  film  catalogs. 

1.  Preview  films  to  be  sure  that  they  contribute  to  the  fulfillment  of  the 
learning  objectives. 

2.  Introduce  the  film  and  specify  learning  objectives. 

3.  Allow  time  for  the  participants  to  discuss  the  film  after  viewing  it. 

Movies,  Training  Films,  and  Audiovisual  Aids. 


For  More  Information: 

Bergevin,  P.,  D.  Morris,  and  R.  M.  Smith.  Adult  Education  Procedures:  A  Handbook  of  Tested  Patterns  for 
Effective  Participation.  New  York:  The  Seabury  Press,  1963. 

Brown,  James  W„  Richard  B.  Lewis,  and  Fred  F.  Harcleroad.  AV  Instruction:  Media  and  Methods.  New 
York:  McGraw-Hill,  1973. 

Otto,  Calvin  P.,  and  Rollin  0.  Glaser.  The  Management  of  Training:  A  Handbook  for  Training  and 
Development  Personnel.  Reading,  MA:  Addison-Wesley,  1970. 

Pula,  Fred  John.  Application  and  Operation  of  Audiovisual  Equipment  in  Education.  New  York:  John 
Wiley  &  Sons,  Inc.,  1968. 

Rigg,  Robinson  P.  Audiovisual  Aids  and  Techniques  in  Managerial  and  Supervisory  Training.  London: 
Hamish  Hamilton,  Ltd.,  1969. 


Instructional  Television 


Instructional  television  is  comparable  to  film  in  its  uses  and  advantages.  An  effective  television 

presentation  brings  the  action  to  the  viewer;  trainees  have  a  chance  to  observe  things  which  would  be  very 

difficult  to  replicate  in  a  classroom.  Like  other  means  of  audiovisual  presentation,  television  can  be  used  in 

individual  instruction,  particularly  when  the  tapes  are  in  video  cassette  form. 

How  It  Can  Be  Used:  1.  To  record  a  role  play,  discussion,  or  presentation  and  replay  it  for  detailed 

examination. 

2.  To  record  a  session  for  viewing  by  persons  unable  to  attend. 

3.  To  demonstrate  desirable  performance  in  a  more  realistic  setting  than  the 
classroom. 

4.  To  take  the  place  of  the  instructor  when  he/she  cannot  be  present. 

Who  Can  Benefit  from 

It:  Supervisors  and  managers  at  all  levels. 

Advantages:  1.  Television  can  take  management  training  out  of  the  academic  realm  and 

place  it  in  an  environment  which  more  closely  resembles  the  real  world. 

2.  Presentations  can  be  taped  ahead  of  time;  trainees  can  replay  parts  of  the 
tape  as  often  as  needed. 

3.  In  the  form  of  video  cassettes,  television  presentations  are  more  convenient 
and  often  less  costly  to  produce  than  films. 

4.  Video  cassettes  are  more  adaptable  to  small  groups  than  is  film. 

5.  Video  cassettes  can  be  set  up  and  operated  more  easily  than  film  can. 

6.  Most  trainers  can  handle  the  mechanics  of  showing  a  television  presentation; 
thus  it  is  easy  for  them  to  tailor  a  presentation  to  the  needs  of  a  particular 
group. 

Disadvantages:  1.  The  videotape  player  and  monitor  are  more,  ex  pensive  than  a  film  or  a  slide 

projector  and  screen. 

2.  The  viewing  area  is  limited  to  the  size  of  the  monitor.  It  may  be  too  small  to 
be  seen  by  large  groups. 

Materials:  1,  Video  cassette  tapes  for  individual  instruction  are  available  from  training 

media  sources. 

Procedures:  1.  Preview  prerecorded  tapes  as  you  would  a  film. 

2.  Introduce  the  tape  and  go  over  the  learning  objectives. 

Related  Approaches:  Closed  Circuit  TV,  Audio  Cassette  Tapes,  Programmed  Instruction,  Audiovisual 

Aids. 

For  More  Information: 

Brown,  James  W.,  Richard  B.  Lewis,  and  Fred  F.  Harcleroad.  AV  Instruction:  Media  and  Methods.  New 
York:  McGraw-Hill,  1973. 

Otto,  Calvin  P„  and  Rollin  O.  Glaser.  The  Management  of  Training:  A  Handbook  for  Training  and 
Development  Personnel.  Reading,  MA:  Addison-Wesley,  1970. 

Pula,  Fred  John.  Application  and  Operation  of  Audiovisual  Equipment  in  Education.  New  York:  John 
Wiley  &  Sons,  Inc.,  1968. 

Rigg,  Robinson  P.  Audiovisual  Aids  and  Techniques  in  Managerial  and  Supervisory  Training.  London: 
Hamish  Hamilton,  Ltd.,  1969. 
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Self-Paced  Instiuction 

Self-paced  instruction  is  a  training  method  in  which  the  content  is  presented  by  means  of 
programmed  study  materials  rather  than  by  an  instructor.  It  can  be  used  by  an  individual  or  integrated  with 
group  instruction.  Programs  have  the  following  features: 

1.  The  material  to  be  presented  is  carefully  and  logically  organized. 

2.  The  content  of  the  program  is  presented  in  segments  or  exercises.  Before  the  learner  can 
proceed  to  the  next  segment,  he/she  must  respond  to  the  first  one. 

3.  Learners  find  out  immediately  whether  they  have  mastered  each  segment  of  material. 

Self-paced  instruction  uses  a  variety  of  media,  including  books,  audio  tape,  slides,  films  and  television. 
Quite  often,  combinations  of  media  are  utilized.  Programs  are  prepared  according  to  the  following 
principles: 

1.  The  program  is  designed  to  accomplish  a  set  of  objectives.  These  objectives  are  precise 
statements  of  the  behavior  to  be  learned  by  the  student.  Objectives  are  derived  from  an 
analysis  of  the  tasks  to  be  performed  on  the  job. 

2.  The  program  is  tested.  If  the  learners  do  not  score  well  on  criterion  test  items  or  exercises,  the 
program  is  revised  and  retested.  This  process  continues  until  a  sampling  of  the  target 
population  is  able  to  complete  the  program  at  a  predetermined  level  of  achievement. 

How  It  Can  Be  Used:  1 .  To  teach  principles,  concepts,  and  techniques  that  do  not  require  practice  or 

demonstration. 

2.  To  give  learners  the  information  they  may  need  as  background  if  they  are  to 
profit  from  such  other  techniques  as  group  discussion,  role  playing,  and  case 
studies. 

With  appropriately  developed  programs,  self-paced  instruction  can  be  used  for  all 
levels  of  supervisors  and  for  mid-level  managers. 

1.  This  method  can  be  adapted  to  the  individual's  needs  and  to  the  amount  of 
time  that  is  available  for  training. 

2.  Because  so  many  media  and  sources  of  information  can  be  used,  the  training 
content  for  self-paced  instruction  is  almost  unlimited. 

3.  Getting  immediate  feedback  on  what  has  been  learned  increases  the  learner's 
confidence  and  motivation. 

1.  Self-paced  learning  should  be  supplemented  by  other  training  approaches  if 
practice  in  applying  the  learning  is  needed. 

2.  Whether  the  training  is  used  on  the  job  depends  on  the  motivation  of  the 
individual  learner. 

3.  The  cost  of  developing  self-paced  instructional  materials  can  be  high. 

4.  Developing  self-paced  instruction  involves  analysis  and  specification  of 
behavioral  objectives.  It  also  involves  designing,  testing,  and  retesting 
materials.  This  requires  more  time,  effort,  and  talent  than  many  other 
methods  of  instruction. 

Procedures:  1.  Before  the  learner  begins  studying  the  material,  he/she  is  introduced  to  both 

the  material  and  any  equipment  that  he/she  may  have  to  use.  If  the  course  is 
taught  in  a  learning  center  (LC),  the  LC  coordinator  responds  to 
student-initiated  requests  for  information  and/or  clarification,  observes 
progress,  and  provides  necessary  assistance.  If  subject  matter  experts  are 
available,  the  coordinator  will  arrange  for  them  to  be  available  to  the  learner 
if  and  when  needed. 

2.  Self-paced  instruction  can  be  integrated  with  conventional  group  methods: 
after  completion  of  a  segment  of  materials  the  instructor  conducts  group 
discussion  designed  to  reinforce  participant  learning  gained  through  use  of 
the  programmed  material. 

Programmed  Instruction,  Personalized  Self-Instruction,  Programmed  Learning. 


Who  Can  Benefit  from 
It: 

Advantages: 

Disadvantages: 


Related  Approeches: 
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Coaching 

Coaching  usually  involves  a  one-to-one  relationship  between  the  “coach"  (a  supervisor  or  manager 
who  is  more  experienced  or  at  a  higher  level  than  the  person  being  coached)  and  the  understudy,  but 
coaching  sessions  can  be  held  for  groups  as  well.  Besides  the  obvious  benefits  to  the  understudy's 
professional  development,  the  time  spent  working  closely  with  a  superior  often  makes  the  participant  more 
confident  about  his  or  her  own  abilities  and  more  comfortable  about  dealing  with  other  superiors.  Coaching 
is  an  ideal  form  of  training  for  supervisors  and  managers  in  that  they  gain  experience  in  dealing  with 
situations  as  they  arise,  yet  there  is  constant  supervision  from  a  knowledgeable  coach. 

How  It  Can  Be  Used:  1.  To  promote  the  interchange  of  information  between  different  levels  of 

supervision  or  management. 

2.  To  prepare  an  understudy  to  assume  a  particular  position. 

3.  To  continue  particular  managerial  philosophies  or  approaches. 

Who  Can  Benefit  from 

It:  1.  Newly  appointed  supervisors  and  managers. 

2.  Persons  chosen  to  assume  the  same  position  as  the  coach's  but  in  another 
location. 

Advantages:  1.  The  coaching  can  be  geared  specifically  to  an  individual's  needs  and 

capacities. 

2.  Contact  with  the  coach  is  nearly  continuous. 

3.  The  learner  has  the  benefit  of  both  his/her  own  experience  and  that  of  the 
coach. 

Disadvantages:  1.  How  valuable  the  training  is  depends  to  a  large  extent  on  the  coach's 

competence  as  a  coach.  -  - 

2.  It  is  sometimes  difficult  to  get  supervisors  or  managers  at  higher  leve's  to 
devote  the  large  amount  of  time  needed  for  good  coaching. 

3.  In  some  instances,  the  understudy  is  merely  an  observer  or  mimic  of  the 
coach's  behavior. 

Procedures:  t.  The  coach  and  the  understudy  agree  on  what  will  be  required  of  each  during 

the  coaching. 

2.  The  understudy's  learning  needs  are  prioritized  and  the  coach  is  advised  of 
the  priorities. 

3.  If  desired,  a  written  training  plan  can  be  worked  out. 

4.  The  training  office  assists  by  planning  learning  objectives,  by  supplying 
developmental  materials  such  as  reading  lists,  and  by  conducting  special 
developmental  activities. 

Related  Approaches:  Understudy. 

For  More  Information: 

Craig,  Robert  L.  (Ed.).  Training  and  Development  Handbook  (Sponsored  by  ASTD)  New  York 

McGraw-Hill,  1976. 

Kirkpatrick,  Donald  L.  A  Practical  Guide  for  Supervisory  Training  and  Development  Reading,  MA 

Addison-Wesley,  1971. 

Singer,  Edwin  J.  Effective  Management  Coaching.  London:  Institute  of  Personnel  Management,  1974. 
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Planned  Exercises 
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Planned  developmental  experiences  offer  the  advantages  that  job  rotation  and  coaching  offer,  but 
most  of  them  do  not  require  that  the  employee  leave  his  or  her  present  position  or  have  the  continuous 
attentior  of  a  superior,  A  well  designed  program  of  planned  experiences  can  have  the  same  results  as 
coaching  or  job  rotation.  Some  types  of  planned  experiences  are: 

Special  assignments  or  special  projects  (for  example,  a  study  to  determine  the  advisability  of  setting  up  a 
new  organizational  unit).  These  assignments  can  be  used  equally  well  for  developing  employees'  skills  in 
specific  areas  of  interest  or  for  familiarizing  them  with  the  overall  operations  of  the  organization. 

Work  on  committees  or  task  forces.  This  provides  experience  in  working  with  others  and  sometimes  in 
researching  and  preparing  reports.  Often  employees  make  important  contacts  with  others  in  the  field 
through  such  work. 

Membership  in  professional  societies,  advisory  groups,  etc.  While  representing  his  or  her  organization  at 
meetings,  the  participant  is  promoting  professional  contacts  and  is  keeping  up  on  developments  in  the 
profession. 

Assignments  as  acting  manager  or  supervisor.  This  is  an  excellent  way  to  fill  the  position  of  a  temporarily 
.ib'ent  supervisor  or  manager  while  training  the  subordinate  at  the  same  time.  The  subordinate  should  be 
-hi  full  responsibility  when  possible. 


How  They  Can  Be  Used:  1.  To  provide  experience  that  the  employee  cannot  get  in  his  or  her  current 

position. 

2.  To  enhance  the  participant’s  abilities  in  the  areas  of  problem  solving, 
cooperation,  and  advocacy. 


Who  Can  Benefit  from 
Them: 


Advantages: 


Disadvantages: 


Procedures: 


1.  Depending  on  the  experiences  provided,  all  levels  of  supervision  or 
management  can  profit  from  them. 

2.  This  approach  is  especially  beneficial  for  persons  whose  present  jobs  are 
somewhat  narrow. 

1.  Assignments  that  allow  for  new  experiences  help  supervisors  and  managers 
to  assess  their  own  performance  in  different  situations;  they  also  can 
provide  indications  of  competencies  in  other  areas. 

2.  Trainees  usually  gain  from  the  outside  experiences  skills  that  help  them  in 
their  regular  jobs. 

3.  A  person's  development  will  be  more  rounded  if  he/she  has  had  a  variety  of 
planned  experiences. 

1 .  Whether  an  individual’s  developmental  objectives  can  be  met  depends  on  the 
availability  of  appropriate  experiences. 

2.  Special  assignments  must  be  made  carefully  so  as  not  to  infringe  on  the 
responsibilities  of  other  organizational  units. 

3.  An  assignment  may  "get  the  job  done"  but  not  yiold  a  planned  experience. 

1.  The  training  office  can  suggest  possible  experiences  to  the  supervisor  of 
persons  for  whom  the  experiences  would  be  appropriate. 

2.  Information  on  past  assignments  and  the  participant's  reactions  to  them 
should  be  kept  by  the  training  office. 


Related  Approaches:  Planned  experiences  are  also  referred  to  at  Planned  Individual  Development  and 

Outside  Assignments. 

For  More  Information: 

Byers,  Kenneth  T.  (Ed.).  Employee  Training  and  Development  in  the  Public  Sector.  Chicago:  International 
Personnel  Management  Association,  1974. 

Tracey,  William  R.  Designing  Training  and  Development  Systems.  New  York:  American  Management 
Association,  Inc.,  1971. 
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In  this  type  of  training,  the  supervisor/manager  usually  receives  the  necessary  study  materials 

through  the  mail.  Working  at  his  or  her  own  pace,  the  student  completes  lessons  one  at  a  time  and  mails  the 

responses  to  test  items  to  a  correspondence  center.  The  responses  are  quickly  graded  (often  by  machine) 

and  returned.  Several  lessons  make  up  a  course  or  learning  module. 

How  It  Can  Be  Used:  1 .  To  teach  cognitive  skills. 

2.  To  introduce  basic  information  which  can  be  expounded  upon  later  in 
face-to-face  instruction. 

Who  Can  Benefit  from 

It:  1.  Entry-level  management  and  supervisor  trainees. 

2.  Managers  Shd  supervisors  who  want  to  familiarize  themselves  with  new 
concepts  or  techniques. 

Advantages:  1.  This  approach  provides  for  the  regular  evaluation  of  individual  learning 

achievement  throughout  the  course. 

2.  The  employee's  work  is  not  necessarily  interrupted  since  studying  is  not 
usually  done  during  work  hours. 

3.  Because  there  are  no  additional  expenses  beyond  those  of  the  tuition  and 
the  materials,  correspondence  study  is  often  less  expensive  than  other 
training. 

4.  If  employees  cannot  conveniently  be  assembled,  correspondence  courses  can 
be  used  to  provide  the  same  lessons  to  each  of  them. 

Disadvantages:  1.  Prepared  correspondence  courses  which  address  the  particular  needs  of 

managers  and  supervisors  may  not  be  available. 

2.  Instructional  objectives  for  this  method  are  limited  to  those  for  which 
competency  can  be  demonstrated  through  responses  to  a  paper-and-pencil 
test. 

3.  Correspondence  courses  must  be  supplemented  by  other  training  approaches 
when  the  training  required  involves  developing  interaction  skills  or  similar 
participatory  skills. 

4.  It  is  often  difficult  to  prepare  "student-motivating"  instructional  com¬ 
ponents. 

5.  Completion  rates  are  significantly  lower  than  those  for  other  types  of 
training. 

Materials:  1.  Emphasis  should  be  placed  on  selecting  courses  with  the  appropriate 

objectives  since  the  course  usually  cannot  be  modified  to  account  for 
changes  in  subject  matter  or  for  the  needs  of  individual  learners. 

2.  Course  materials  and  strategy  should  be  validated  for  the  I  earner's 
characteristics  and  learning  environment.  This  is  especially  important  since 
the  student  works  alone. 

Procedures:  1.  The  person  who  arranges  for  the  course  should  verify  that  all  of  the 

instructional  materials  are  available  to  the  learner;  some  correspondence 
courses  require  reference  material  and  audiovisual  equipment  not  supplied 
with  the  lessons. 

2.  The  student  must  be  told  the  objectives,  the  criteria  for  success,  and  the 
consequences  of  late  submissions,  wrong  answers,  etc. 

3,  If  possible,  the  student  should  have  an  opportunity  to  contact  an  instructor 
(perhaps  by  telephone)  to  ask  questions,  get  feedback  on  performance,  etc. 

Related  Approaches:  Extension  Courses,  Home  Study. 

For  More  Information: 

Craig,  Robert  L.  (Ed.).  Training  and  Development  Handbook  (Sponsored  by  ASTD).  New  York: 

McGraw-Hill,  1976. 

Erdos,  Renee  F.  Teaching  by  Correspondence  (A  UNESCO  Source  Book).  London:  Longmans,  Green  &  Co. 

Limited,  1967. 
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APPENDIX  D 

ANALYSIS  OF  QUESTIONNAIRE  RESPONSES 

Introduction 

As  is  discussed  in  the  Study  Methodology  section  of  this  report,  a 
questionnaire  was  developed  to  obtain  trainee  impressions  of  the  courses  they 
attended.  The  16  questionnaires  (13%  return)  received  were  a  disappointingly 
small  response.  They  provide  too  little  data  to  be  definitive.  Discussion 
of  the  responses  was,  therefore,  excluded  from  the  main  report  and  appears 
in  this  Appendix. 

The  Questionnaire 

A  copy  of  the  3-page  questionnaire  is  given  on  pages  91-93  •  The  original 
was  on  legal-size  paper. 

From  a  question  design  standpoint,  DTIC  may  wish  to  consider  the  rating 
scale  format  used  for  questions  2  and  9  in  an  evaluation  instrument.  Provided 
that  the  number  of  options  is  not  too  large  to  obliterate  distinctions,  people 
usually  find  this  format  more  adaptable  to  the  assessments  they  want  to  express 
than  a  more  restrictive  format.  For  example,  Part  III  of  DDC  Form  345  might 
be  amenable  to  restructuring. 

The  Responses 

Tables  1  and  2  summarize  the  responses  to  Questions  1  and  14.  Not 
unexpectedly,  usage  of  the  Lockheed,  SDC,  and  BRS  systems  correlates  with 
training  courses  that  respondents  attended.  The  advanced  courses  appear 
almost  as  popular  as  the  introductory  courses.  Question  14  responses  indicate 
a  popularity  of  the  New  York  Times  database.  The  distribution  by  year  of  the 
training  experiences  indicates  that  perhaps  half  of  the  respondents  have  3 
and  4  years  of  searching  experience,  and  almost  all  appear  to  have  been 
searching  for  at  least  one  year.  Responses  to  Question  14  showed  moderately 
heavy  multiple  database  usage.  Five  respondents  use  5  or  more  of  the  online 
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systems,  and  4  use  3  of  the  systems. 

It  is  difficult  yet  to  know  whether  the  usage  reported  in  Questions  13 
and  14  represent  a  low  potential  utilization  of  online  database  or  a  level 
that  accords  with  end-user  need.  Current  weekly  usage  of  online  systems  is 
shown  in  Table  3. 

The  prevalent  system/database  use  of  0-2  hours/day  reported  in  Question 
14  suggests  one  group  of  searchers  who  average  3/4  to  1  hour  of  terminal 
activity  daily  and  another  group  who  work  at  terminals  about  3  hours  daily. 

A  special  announcement  was  made  at  the  Washington,  D.C.,  Online  Users 
Group  meeting  for  questionnaire  responses  in  the  hope  that  a  good  response 
could  be  obtained  from  federal  librarians.  Table  4  shows  that  this  didn't 
happen.  Over  half  of  the  respondents  are  from  the  for-profit  sector. 

The  most  popular,  wanted  teaching  methodologies  reported  for  Question 

3  are: 

Lecture  (12  votes) 

Discussion  (6) 

Transparencies  (6) 

Hands-on  practice  (4) 

The  two  most  disliked  methodologies  are: 

Role  playing  (4) 

Lecture  only  (3) 

Some  who  voted  positively  for  lectures  specified  their  preference  for  lectures 
combined  with  online  demonstrations  on  CRTs  visible  to  the  students.  One  person 
considered  such  lectures  "ruined"  when  the  computer  system  went  down.  Having 
experienced  this  unsatisfying  situation,  the  author,  when  teaching,  developed 
handout  and  transparency  materials  as  backups  and  recommends  this  precaution 
to  DTIC  in  this  report. 

Open-ended  questions  such  as  4,  5  and  6  are  a  risky  way  of  trying  to 
obtain  information  because  people  frequently  ignore  them.  Only  2  people  did 
not  respond  in  some  way  to  Question  4,  The  program  content  mentioned  the  most 

■I 
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was  hands-on  practice  (6  times)  and  search  strategy  examples  (4  times).  Other 
course  content  respondents  found  helpful  included: 

-  Exchange  of  experiences  with  other  participants 

-  Explanation  of  how  the  computer  system  executes  various 

commands 

-  Discussion  of  strategies  appropriate  for  assigned 

problem  questions 

-  Database  design 

-  Explanation  of  databases  in  the  context  of  applying 

search  questions  to  them 

-  Indexing  policies  (in  advanced  courses) 

More  hands-on  practice  was  the  most  frequently  mentioned  suggestion 
(4  times)  of  program  content  respondents  did  not  consider  adequate. 

Three  respondents  criticized  the  "sales  pitch"  content  of  courses,  one 
suggesting  that  courses  be  taught  by  experienced  searchers,  not  sales  persons. 
Others  wanted  more  of  the  following  content: 

-  Comparison  of  various  systems  (differences,  advantages) 

-  Discussion  of  real-life  problems,  how  they  occur  and 

are  solved 

-  System  documentation 

-  Better  use  of  transparencies  (removing  them  too  fast 

for  note  taking  criticized) 

Only  7  respondents  provided  information  for  open-ended  Question  6. 

That  this  question  was  passed  over  might  have  been  predicted  because  it  is 
more  difficult.  It  asks  people  to  reach  beyond  their  experience  and  into 
their  imagination.  Content  suggestions  included: 

-  The  rationales  underlying  the  structures  of  databases 

-  System  and  file  design  (linear  and  inverted  files, 

stopword  list,  string  characteristics,  etc.) 

-  Subscription  price  information 

-  Hands-on  practice 

-  Difficult  multi-term  searches 

-  Management  issues 

-  Historical  (this  may  refer  to  the  development  of 

online  services  or  the  evolution  of  particular  databases) 


Respondents  provided  a  r  nge  of  1  to  16  hours  as  needed  to  gain  a  working 
knowledge  of  a  single  database,  and  a  range  of  2  hours  to  4  days  for  two  related 
databases  (Question  7).  This  is  a  type  of  question  for  which  computation  of 
an  arithmetic  mean  would  have  little  meaning.  Of  the  people  who  reported 
attending  3  to  8  training  courses,  8  individuals  specified  8  or  more  hours  for 
a  single  database.  Within  this  group,  4  people  who  had  attended  6  to  8  courses 
specified  a  training  period  of  2  or  more  days  for  2  related  databases.  It  is 
believed  that  more  weight  should  be  given  to  opinions  of  those  with  more 
course  experience. 

Table  5  shows  the  responses  to  Question  8.  Respondents  were  about 
evenly  divided  between  the  sufficiency  of  one  course  and  a  two-course  sequence 
provided  that  updates  on  system  +  database  combinations  are  available.  This 
response  underscores  the  value  found  in  updates  end  the  importance  of  making 
them  real  learning  experiences. 

Remarks  provided  for  Question  10  reinforced  those  for  Questions  3,  4 
and  5.  Five  respondents  addressed  content  rather  than  type  of  teaching  material, 
mentioning  search  strategy  examples,  case  studies,  prototype  searches,  and 
search  studies  such  as  have  appeared  in  the  periodicals,  Pat  abase  and  Online. 
Other  responses  recommending  the  following  materials: 

-  Good  manuals  (3  mentions) 

-  Handouts  sufficiently  comprehensive  to  alleviate 

note  taking  (3  mentions) 

-  Transparencies 

-  Audiovisual  modules 

-  Charts 

-  Venn  diagrams 

-  Hands-on  quizzes 

-  Quick  reference  pamphlets /cards 

Respondents  were  negative  about  self-teaching  manuals  and  CAI  replace¬ 
ments  for  more  conventional  methodologies.  For  self-teaching  manuals,  they 
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TABLE  3 

13.  Bow  Much  online  searching  do  you  do? 

Flense  circle  Che  appropriate  letter  below. 

__L  *•  °ver  20  hre/wk  4  d.  5-10  hr«/wk 

JbL  k.  15-20  hrs/wk  I  «.  3-5  hrs/wk  No  response 

c.  10-15  hre/wk  3  f.  %-3  hre/wk 

O  |.  don't  do  searching 


TABLE  4 


15.  What  type  of  organ  lest  Ion  do  you  work  for? 

3  A.  For  profit  (not  e  DOD  contractor)  JLf. 
^  b.  For  profit  (DOD  contrector)  Q  g. 
A  «.  For  profit  (Information  broker)  O  h. 
oE  4.  Mot  for  profit  (non-ecedemlc)  O  l. 
3  e.  Mot  for  profit  (academic) 


Federal  govt,  (non-cllltary) 
Federal  govt,  (military) 
State,  local  government 
Self  employed 


TABLE  5 


8.  Which  of  the  following  altematlvee  do  you  consider  the  beet  to  acquire 
searching  proficiency?  (Please  circle  the  appropriate  letter) 


O 


•,  *  single  couree  followed  by  on-the-job  practice 

b.  A  beginning  course,  on-the-job  practice;  6-12  months  later,  an  advanced  course 


C.  A  beginning  course,  on-the-job  practice;  6-12  months  later,  an  advanced 
course  on  the  command  language  and  system  and  advanced  database  courses 
d.  Alternative  a  and  periodic  1-day  updates  on  system  +  databases 


«.  Alternative  b  end  periodic  1-day  updatas  on  system  *  databases 


f.  Alternative  c  and  periodic  1-day  updates  on  system  +  databases 


c 


4  Yes 
6  No 


5  Maybe 
1  No  response 
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One  person’s  "maybe"  was  accompanied  by  the  note  "when  money  is  short."  To 
CAI,  they  voted 

2  Yes  2  Maybe 

11  No  1  No  response 

It  is  likely  that  at  least  part  of  the  above  negativity  is  caused  by  a  lack  of 
familiarity.  I  had  personally  thought  that  CAI  would  not  be  an  adequate 
substitute  for  the  lecture  mode  until  I  used  it  (on  a  system  previously 
unfamiliar  to  me).  The  experience  indicates  that  CAI  speeds  learning  of  simple 
facts  but  it  cannot  be  used  for  situations  where  students  are  likely  to  ask  a 
lot  of  different  questions  and  need  tailored  answers  to  learn.  CAI  could  be 
used  to  learn  basic  aspects  of  a  command  language  and  database  structure;  its 
present  drawback  is  the  programming  expense.  People  seem  to  fear  the  loss  of 
contact  with  a  human  tutor. 

Hoped  for  information  about  searchers,  search  behavior,  learning  needs 
and  preferences,  and  database  usage  did  not  materialize  from  this  questionnaire 
survey.  The  questionnaire,  as  a  questionnaire,  seems  to  have  passed  a  field 
test  in  that  respondents  seemed  to  have  little  difficulty  understanding  and 
answering  the  questions.  Write-ins  for  Questions  1  and  14  point  to  the  need 
to  add  to  the  named  courses  and  systems  should  these  questions  be  asked  again. 
DTIC  might  find  is  desirable  to  obtain  information  about  the  experiences  of 
dialup  trainees  before  they  receive  DTIC  training.  A  modification  of  Questions 
l  and  14  together  with,  possibly.  Question  4  and  Question  13  could  provide 
this  information.  Knowing  the  experience  trainees  bring  to  DTIC’ s  courses 
might  aid  the  interpretation  of  responses  to  DTIC's  Course  Evaluation  ques¬ 
tionnaire  (DDC  Form  345). 
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programs  of  the  following  organizations  for  examination:  (a)  Lockheed  Information! 
Retrieval  Service  (LIRS)  (2  programs),  (b)  University  of  Denver  Graduate  School 
of  Librarianship  &  Information  Management  (GSLIM),  (c)  National  Library  of  Medi¬ 
cine  (NLM),  (d)  University  of  Dayton  Research  Institute  (UDRI),  (e)  University  off 
Pittsburgh  On-Line  Training  Center  (Pitt),  (f)  House  Information  Systems  (H.I.S. )| 
(g)  Congressional  Research  Service,  Library  of  Congress  (CRS),  and  (h)  Defense 
Technical  Information  Center  (DTIC). 

A  set  of  criterion  variables  was  developed  by  which  the  instruction  programs 
were  analyzed  and  compared.  These  variables  address  four  major  areas:  (1)  pur¬ 
posive  documentation  (goals,  objectives,  criteria),  (2)  education  program  (compo-j 
nents,  duration,  availability,  location,  participants),  (3)  resources  (human, 
equipment,  facilities,  instruction  materials),  and  (4)  evaluation/f eedback  mech¬ 
anisms  (instruments,  tests,  follow-ups,  user  groups). 

Most  programs  are  found  to  be  weak  in  purposive  documentation  that  limits  LheJ 
bases  on  which  their  developers  can  evaluate  their  accomplishments.  Five  of  the 
programs  rely  on  lectures  as  the  principal  mode  of  knowledge  transfer.  Pitt  and 
H.I.S.  programs  integrate  lecturing  with  hands-on  practice.  LIRS's  ONTAP  and  CRS| 
programs  provide  online  self-instruction.  Four  of  the  programs  have  a  multi- 
course  structure,  offering  one  or  more  advanced  courses  beyond  the  introductory 
one.  No  course  other  than  the  academic  one  (GSLIM)  is  longer  than  one  week  in 
duration.  Only  LIRS's  ONTAP  program  is  substantially  location  independent;  all 
others  require  students  to  travel  to  either  a  single  location  or  one  of  several 
locations.  Although  six  of  the  courses  have  no  course  fees,  the  high  indirect 
costs  of  travel  probably  prevent  some  individuals  needing  instruction  from  obtain), 
ing  it. 

Approximately  1  to  1^  months  appears  to  be  needed  by  an  instructor  to  plan  a 
program  and  prepare  printed  instruction  materials  for  each  content  focus,  i.e.,  a 
command  language  or  a  database.  Preparation  time  for  two  or  more  foci  appears  to 
be  cumulative,  for  example,  2-4  months  for  a  command  language  and  a  database. 

Complex  databases  require  more  time,  for  example,  3  months  for  a  UDRI  manual. 

Time  needed  to  revise  a  program  and  its  instruction  materials  can  vary  from  3 
days  to  1  month  dependent  on  the  extent  of  the  revision.  By  comparison,  instruc-| 
tors  can  spend  10  to  100  hours  to  prepare  one  hour  of  computer-assisted  instruction. 

Only  the  Pitt,  ONTAP,  and  CRS  programs  provide  each  student  with  a  terminal; 
in  most  other  programs,  two  people  share  a  terminal.  Irrespective  of  the  instrucj- 
tion  methodology  employed,  all  programs  produce  printed  materials  for  students. 

These  vary  from  very  brief  guides  to  a  database  and  retrieval  options  (H.I.S.)  to) 
replicas  of  entire  lecture  content  (NLM). 

Six  of  the  programs  distribute  questionnaires  to  students  at  the  end  of  a 
program  to  obtain  a  course  evaluation.  No  program  has  a  longer  term  evaluation 
mechanism,  for  example,  a  follow-up  six  months  after  the  course.  The  customer 
service  desks  maintained  by  LIRS,  DTIC,  NLM,  and  H.I.S.  provide  feedback  to  in¬ 
structors,  but  only  for  NLM  is  this  direct  since  NLM  instruction  staff  also  per¬ 
form  service  desk  duties.  The  user  group  meetings  of  DTIC  and  NLM  and  the  update| 
session^  of  LIRS  are  other  opportunities  for  feedback. 

t  t  ^fj^s“on  practice  is  a  sine  qua  non  for  these  instruction  programs.  Practice! 

°  a  n§  4  to  5  hours  per  student  is  suggested  for  programs  having  the  objective 
of  providing  the  student  with  proficiency  to  do  online  searches  on-the-job. 

CRS's  computer-based  instruction  (CBI)  program  demonstrates  that  the  CBI 
methodology  can  be  used  to  convey  introductory- level  knowledge  of  a  corraaand  lan¬ 
guage  and  databases.  CBI  is  not  yet  hospitable  to  teaching  advanced  search  com¬ 
plexities.  It  is  also  expensive.  The  LIRS  ONTAP  approach,  that  uses  computers 
in  a  more  limited  way,  is  worthy  of  further  study.  It  is  self-instructional  and,| 
with  proper  printed  materials,  it  might  be  able  to  effectively  replace  up  to  85% 
of  present  instruction  programs. 
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